From 8290c81eea38db28e10cb2d56aa6ac87ced92cb6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=A1=BE=E7=9B=BC?= Date: Tue, 1 Sep 2026 06:37:23 +0000 Subject: [PATCH] =?UTF-8?q?feat(qwen-live):=20standalone=20voice=20daemon?= =?UTF-8?q?=20package=20=E2=80=94=20M1=20+=20M2=20(#10367)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(qwen-live): scaffold standalone live daemon package with ported host stack New workspace package @qwen-code/qwen-live (M1+M2 of the Live split, issue #10118). This commit adds the package skeleton and the verbatim ports that need no behavioral changes: - host/: Live Host protocol v6 types, LiveHostCoordinator, and the discovery-file writer, ported from packages/cli/src/serve/live/ together with their test suites (51 tests green) - realtime/socket.ts, realtime/sanitize.ts: the two helper symbols the realtime client depends on, extracted from cli ui utils - registered in scripts/build.js buildOrder and root vitest projects * feat(qwen-live): generalize the realtime client tool surface Port qwen-realtime-session.ts and adapt it for the orchestration model: tools and instructions come from config, every declared tool dispatches through onFunctionCall with the raw argument payload, and outputs are receipt-style via submitFunctionOutput. The single-handoff machinery (multiple_handoffs error, steering acknowledgement, open-call complete) is removed; transport, response arbitration, pendingCalls timing, barge-in, and defensive validation are unchanged from the port. capturesTranscript marks handoff-style tools that capture the live transcript tail at call time. Test suite adapted: 27 tests green. * feat(qwen-live): add the BackendAdaptor seam and the qwen serve adaptor BackendAdaptor is the capability-bit-driven port between the live orchestrator and a coding-agent backend. QwenCodeAdaptor drives a running qwen serve daemon through the @qwen-code/sdk DaemonClient: 202-admission prompts, SSE event normalization (chunks and tool calls live under session_update.data.update), reject-if-idle mid-turn steering with a plain-prompt fallback, nested permission votes with voter attribution, and attachment-backed image blocks. 23 unit tests against an injected fake client. * feat(qwen-live): orchestrator, tools, injector, permissions, and daemon entry The M1+M2 core: - seven receipt-style dispatch tools plus remain_silent, with a dispatcher that answers every call (timeout fallback included) and session_N/job_N/asset_N handle registries - LiveSession wires Host audio, the realtime connection, tool handlers (appshot via the Host capture channel, handoff with voice-context packing and image attachments, busy-session steering), backend event pumps, and a bounded graceful-stop drain - Injector: safe-insertion window (VAD + response-in-flight + playback estimate at 48 kB/s with a quiet gap), batch aggregation, per-job progress throttling, barge-in drops progress, spoken/detail split - PermissionBroker: allow_always stays a one-shot protocol vote backed by a local TTL rule; remote resolutions retract queued asks - LiveDaemon: Host WS endpoint (Bearer + Origin-reject), discovery publish with fail-fast mutual exclusion and dead-owner handoff, appshot channel marked ready (the in-process capture path needs no separate verification), SIGINT/SIGTERM shutdown - SessionLog: JSONL with synchronous rotation (rename-before-reopen) 116 new unit tests; package total 167 green. * feat(qwen-live): end-to-end test suites against a real qwen serve daemon Test stack: a real qwen serve (spawned CLI bundle, fake-openai model endpoint), a scriptable fake DashScope realtime WebSocket server, the qwen-live daemon as a child process, and a protocol-v6 FakeHost client — every boundary is the real wire. Scenarios (11 tests): discovery publish/remove and hello/welcome, toggle brings up the realtime session (tool schema + instructions asserted on session.update), two-way audio byte fidelity, the full handoff loop against real serve with [COMPLETE] injection and spoken summary, injection-window hold/batch semantics, permission ask/vote relay, and busy-session steering tiers. The suites caught a real bug the unit layer could not: per-session calls must echo the daemon-issued clientId from the create/attach response (a self-made id is rejected by the client registration guard and broke handoff admission and vote attribution). QwenCodeAdaptor now tracks and echoes the issued id per session. * chore(release): keep @qwen-code/qwen-live versioned independently Same treatment as sdk/mobile-mcp/node-repl: the incubating package stays at its own 0.x version instead of lock-stepping with the CLI release train. The verbatim guard in package-scripts.test.js is updated alongside. * fix(qwen-live): harden review findings across dispatcher, injector, and permission relay Self-review fixes bundled after the adversarial pass: - adaptor: classify snake_case permission option ids (proceed_once) — the word-boundary regex never matched through "_", so an allow vote silently degraded to a cancelled outcome and serve dropped the turn - adaptor: detect the pending-prompt limit by error name as well as status, and fall image-bearing steers back to a regular prompt - dispatcher: a timed-out tool now answers with a pending receipt that tells the model not to retry (the handler keeps running) - injector: permission items flush ahead of the size-capped context join; spoken lines merge whole-line greedily under the 280-char cap - live-session: resubscribe the event pump with backoff until session_closed; a rejected permission chain can no longer become an unhandled rejection; receipts are clamped and never empty - permission broker: a failed auto-approval falls back to the spoken ask instead of losing the request - daemon/config/index: appshot readiness is declared at construction, numeric config ports parse, stray rejections are logged not fatal - e2e harness: QWEN_LIVE_E2E_KEEP=1 keeps the temp dirs for post-mortem * fix(qwen-live): address review round 1 — permission voting, shutdown, discovery reclaim, and hardening Permission trust boundary: - classify options by the wire's structured kind enum (allow_once/ allow_always/reject_*) instead of word regexes over snake_case ids, and vote the least-escalating match — a spoken "allow" now lands on proceed_once, never proceed_always_project - respond_permission's note is no longer discarded: it is logged and relayed to the backend session as a follow-up user instruction - track own votes per session so byUs attribution works on adopted sessions; narrow allow_always standing rules to the exact normalized command instead of a two-token prefix Lifecycle: - daemon stop() no longer hangs when SIGINT lands before listen() or when a Host redials during shutdown (stopping gate + client terminate) - dead-owner discovery reclaim now follows the cli contract (no-op commitOwner during handoff, republish after) — previously it could never succeed and a crash left the daemon unable to start - stop drain settles on the input-commit ack instead of burning the full 30s budget; response.created during the drain no longer flips the call back to speaking; a host-initiated start can no longer race deactivate() Robustness: - a late ASR final for an already-consumed input item is ignored as stale_input instead of dropping the call - raise the Host frame cap to cover a conformant CJK accessibility dump (64KiB frame cap contradicted the 32k-char field cap) - bin entry works through npm's bin symlink (realpath + pathToFileURL) - steered handoffs that join a running turn reuse the existing job instead of orphaning it; over-long steers fall back to a queued prompt; session_list unions created cwds, requests a full page, and maps hasActivePrompt for untracked sessions; created-session labels are applied via session metadata - injector: jobless throttle keys use the full context, dedup is symmetric, and barge-in drops clear the throttle stamp; sentence shapers handle CJK punctuation and mid-token periods - config loader: ENOENT-only swallow, BOM strip, ~ expansion, typed port validation, source-accurate errors; session log close() is terminal; transcript retention is capped Tests and docs: new config/daemon/sanitize suites, win32 guards restored on discovery mode assertions, protocol parity test now checks both type copies, qwen-live e2e suites join the no-AK PR gate, fake model gating in the inject e2e matches only the current turn and is consumed on match, FakeHost answers capture requests, lockfile diff reduced to qwen-live-only entries, README states the installer gap. * test(qwen-live): pin injector size caps and steered-message delivery - injector: 6000-char combined-context cap, whole-line spoken join at the 280-char boundary, single-line ellipsis truncation - steering e2e: assert the joined instruction actually reached the model endpoint, not just the receipt wording * fix(sdk): raise daemon browser bundle budget Co-authored-by: Qwen-Coder * feat(qwen-live): scaffold standalone live daemon package with ported host stack New workspace package @qwen-code/qwen-live (M1+M2 of the Live split, issue #10118). This commit adds the package skeleton and the verbatim ports that need no behavioral changes: - host/: Live Host protocol v6 types, LiveHostCoordinator, and the discovery-file writer, ported from packages/cli/src/serve/live/ together with their test suites (51 tests green) - realtime/socket.ts, realtime/sanitize.ts: the two helper symbols the realtime client depends on, extracted from cli ui utils - registered in scripts/build.js buildOrder and root vitest projects * feat(qwen-live): generalize the realtime client tool surface Port qwen-realtime-session.ts and adapt it for the orchestration model: tools and instructions come from config, every declared tool dispatches through onFunctionCall with the raw argument payload, and outputs are receipt-style via submitFunctionOutput. The single-handoff machinery (multiple_handoffs error, steering acknowledgement, open-call complete) is removed; transport, response arbitration, pendingCalls timing, barge-in, and defensive validation are unchanged from the port. capturesTranscript marks handoff-style tools that capture the live transcript tail at call time. Test suite adapted: 27 tests green. * feat(qwen-live): add the BackendAdaptor seam and the qwen serve adaptor BackendAdaptor is the capability-bit-driven port between the live orchestrator and a coding-agent backend. QwenCodeAdaptor drives a running qwen serve daemon through the @qwen-code/sdk DaemonClient: 202-admission prompts, SSE event normalization (chunks and tool calls live under session_update.data.update), reject-if-idle mid-turn steering with a plain-prompt fallback, nested permission votes with voter attribution, and attachment-backed image blocks. 23 unit tests against an injected fake client. * feat(qwen-live): orchestrator, tools, injector, permissions, and daemon entry The M1+M2 core: - seven receipt-style dispatch tools plus remain_silent, with a dispatcher that answers every call (timeout fallback included) and session_N/job_N/asset_N handle registries - LiveSession wires Host audio, the realtime connection, tool handlers (appshot via the Host capture channel, handoff with voice-context packing and image attachments, busy-session steering), backend event pumps, and a bounded graceful-stop drain - Injector: safe-insertion window (VAD + response-in-flight + playback estimate at 48 kB/s with a quiet gap), batch aggregation, per-job progress throttling, barge-in drops progress, spoken/detail split - PermissionBroker: allow_always stays a one-shot protocol vote backed by a local TTL rule; remote resolutions retract queued asks - LiveDaemon: Host WS endpoint (Bearer + Origin-reject), discovery publish with fail-fast mutual exclusion and dead-owner handoff, appshot channel marked ready (the in-process capture path needs no separate verification), SIGINT/SIGTERM shutdown - SessionLog: JSONL with synchronous rotation (rename-before-reopen) 116 new unit tests; package total 167 green. * feat(qwen-live): end-to-end test suites against a real qwen serve daemon Test stack: a real qwen serve (spawned CLI bundle, fake-openai model endpoint), a scriptable fake DashScope realtime WebSocket server, the qwen-live daemon as a child process, and a protocol-v6 FakeHost client — every boundary is the real wire. Scenarios (11 tests): discovery publish/remove and hello/welcome, toggle brings up the realtime session (tool schema + instructions asserted on session.update), two-way audio byte fidelity, the full handoff loop against real serve with [COMPLETE] injection and spoken summary, injection-window hold/batch semantics, permission ask/vote relay, and busy-session steering tiers. The suites caught a real bug the unit layer could not: per-session calls must echo the daemon-issued clientId from the create/attach response (a self-made id is rejected by the client registration guard and broke handoff admission and vote attribution). QwenCodeAdaptor now tracks and echoes the issued id per session. * chore(release): keep @qwen-code/qwen-live versioned independently Same treatment as sdk/mobile-mcp/node-repl: the incubating package stays at its own 0.x version instead of lock-stepping with the CLI release train. The verbatim guard in package-scripts.test.js is updated alongside. * fix(qwen-live): harden review findings across dispatcher, injector, and permission relay Self-review fixes bundled after the adversarial pass: - adaptor: classify snake_case permission option ids (proceed_once) — the word-boundary regex never matched through "_", so an allow vote silently degraded to a cancelled outcome and serve dropped the turn - adaptor: detect the pending-prompt limit by error name as well as status, and fall image-bearing steers back to a regular prompt - dispatcher: a timed-out tool now answers with a pending receipt that tells the model not to retry (the handler keeps running) - injector: permission items flush ahead of the size-capped context join; spoken lines merge whole-line greedily under the 280-char cap - live-session: resubscribe the event pump with backoff until session_closed; a rejected permission chain can no longer become an unhandled rejection; receipts are clamped and never empty - permission broker: a failed auto-approval falls back to the spoken ask instead of losing the request - daemon/config/index: appshot readiness is declared at construction, numeric config ports parse, stray rejections are logged not fatal - e2e harness: QWEN_LIVE_E2E_KEEP=1 keeps the temp dirs for post-mortem * fix(qwen-live): address review round 1 — permission voting, shutdown, discovery reclaim, and hardening Permission trust boundary: - classify options by the wire's structured kind enum (allow_once/ allow_always/reject_*) instead of word regexes over snake_case ids, and vote the least-escalating match — a spoken "allow" now lands on proceed_once, never proceed_always_project - respond_permission's note is no longer discarded: it is logged and relayed to the backend session as a follow-up user instruction - track own votes per session so byUs attribution works on adopted sessions; narrow allow_always standing rules to the exact normalized command instead of a two-token prefix Lifecycle: - daemon stop() no longer hangs when SIGINT lands before listen() or when a Host redials during shutdown (stopping gate + client terminate) - dead-owner discovery reclaim now follows the cli contract (no-op commitOwner during handoff, republish after) — previously it could never succeed and a crash left the daemon unable to start - stop drain settles on the input-commit ack instead of burning the full 30s budget; response.created during the drain no longer flips the call back to speaking; a host-initiated start can no longer race deactivate() Robustness: - a late ASR final for an already-consumed input item is ignored as stale_input instead of dropping the call - raise the Host frame cap to cover a conformant CJK accessibility dump (64KiB frame cap contradicted the 32k-char field cap) - bin entry works through npm's bin symlink (realpath + pathToFileURL) - steered handoffs that join a running turn reuse the existing job instead of orphaning it; over-long steers fall back to a queued prompt; session_list unions created cwds, requests a full page, and maps hasActivePrompt for untracked sessions; created-session labels are applied via session metadata - injector: jobless throttle keys use the full context, dedup is symmetric, and barge-in drops clear the throttle stamp; sentence shapers handle CJK punctuation and mid-token periods - config loader: ENOENT-only swallow, BOM strip, ~ expansion, typed port validation, source-accurate errors; session log close() is terminal; transcript retention is capped Tests and docs: new config/daemon/sanitize suites, win32 guards restored on discovery mode assertions, protocol parity test now checks both type copies, qwen-live e2e suites join the no-AK PR gate, fake model gating in the inject e2e matches only the current turn and is consumed on match, FakeHost answers capture requests, lockfile diff reduced to qwen-live-only entries, README states the installer gap. * test(qwen-live): pin injector size caps and steered-message delivery - injector: 6000-char combined-context cap, whole-line spoken join at the 280-char boundary, single-line ellipsis truncation - steering e2e: assert the joined instruction actually reached the model endpoint, not just the receipt wording * feat(qwen-live): port the Live Host installer — M1 gap closed `live-host-installer.ts` moves verbatim from packages/cli/src/serve/live/ (same OSS/GitHub dual-source manifest fetch, sha256 + codesign + team-identifier verification, staging + backup install with rollback, and the retryable status machine), with its 11-test suite. The daemon gains the matching HTTP surface: - GET /live/setup — installed Host state - POST /live/setup/install — download + verify + install the latest POST /live/setup/launch — open the installed app All three sit behind the daemon's existing Bearer wall (a browser context sending Origin is always refused, wrong token 401) — pinned by a new daemon test; unknown paths stay 404 and GET /healthz answers plain 200. Bootstrap no longer depends on the built-in qwen-code Live integration having installed the Host first. * fix(qwen-live): attach the appshot-timeout rejection handler before advancing fake timers The timeout rejects synchronously inside advanceTimersByTimeAsync; a handler attached only afterwards leaves an unhandled rejection for an instant, which vitest counts as a run-level error (CI fails the package with Errors: 1 even though all 250 tests pass). Attach the expectation first, await it after the clock advances. * test(qwen-live): sync the no-AK gate pin with the qwen-live suites The no-ak-integration-ci pin (added upstream after our branch point) asserts the gate's file list verbatim; add the four qwen-live suites the gate has been running since they joined it. * fix(qwen-live): address review round 2 — trust-boundary keys, backpressure, session lifecycle Criticals (all probed and reproduced before fixing): - broker titleKeyOf: split on the FIRST colon and keep the entire remainder (commands routinely contain colons — a truncated key auto-approved a different -m message), and stop case-folding (/a vs /A are different paths on Linux). The whitespace-collapse pin is preserved - broker deny: an explicit spoken refusal now revokes the standing rule that matched the same request — the silent auto-answer's fallback path could otherwise be overridden by the invisible rule on the next identical request - adaptor activeJobRef: a busy-session fallback (image steer, over- long text 400) no longer overwrites the running turn's ref with the queued prompt's id — later steer receipts report the right job - adaptor byUs attribution: the daemon's originator stamp is now authoritative when present; ownVotes only covers anonymous resolutions. Another client resolving during our vote round-trip no longer reads as our own answer (R1-2 regression, fixed test-first) - adaptor permission titles: control sequences stripped before the title reaches speech or keys a standing rule (whole command kept — no first-line cut, unlike the progress path) - live-session pushAudio: the provider's backpressure signal is propagated (the port source fails the call) and onAudioDropped is wired to failCall — a socket over its buffer cap no longer silently garbles the utterance; the deliberate true for stale/connecting frames is untouched - handles: closed sessions stop resolving entirely (registry-level), and session_closed clears the call's default handle so a session- less handoff rebuilds instead of retrying a dead target forever - handles: read-site reconciliation — session_list and session_monitor transition a session's stale non-terminal jobs to done when the backend itself reports idle (a turn_complete emitted while no pump was subscribed no longer reports a running active_job forever) Suggestions on this PR's new surfaces: - live-host.yml: packages/qwen-live/src/host/** joins the path filter so the parity-test lane runs on qwen-live-only PRs instead of first failing in the merge queue - installer: the install route passes force=true (the recovery path for a corrupted-but-present app, matching the CLI mirror), and launch() gains the non-darwin guard in BOTH copies so the byte parity pin stays green - index: directory-form invocation (node packages/qwen-live) now runs main() instead of exiting 0 silently - realtime: the dead lastHandoffEntryCount machinery is removed (truncate-and-reset replaced the cursor it pretended to track) --------- Co-authored-by: Claude Co-authored-by: 易良 <1204183885@qq.com> Co-authored-by: yiliang114 Co-authored-by: Qwen-Coder --- .github/workflows/live-host.yml | 4 + integration-tests/fake-dashscope-server.ts | 365 +++ integration-tests/qwen-live-harness.ts | 738 ++++++ integration-tests/qwen-live-m1-call.test.ts | 249 ++ integration-tests/qwen-live-m2-inject.test.ts | 234 ++ .../qwen-live-m2-permission.test.ts | 211 ++ .../qwen-live-m2-steering.test.ts | 208 ++ package-lock.json | 128 + package.json | 2 +- .../cli/src/serve/live/live-host-installer.ts | 3 + packages/core/src/tools/tool-registry.test.ts | 6 +- .../src/main/__tests__/protocol.test.ts | 16 + packages/qwen-live/.gitignore | 2 + packages/qwen-live/README.md | 57 + packages/qwen-live/build.mjs | 41 + packages/qwen-live/package.json | 57 + .../src/adaptor/qwen-code-adaptor.test.ts | 1201 +++++++++ .../src/adaptor/qwen-code-adaptor.ts | 837 +++++++ packages/qwen-live/src/adaptor/types.ts | 146 ++ packages/qwen-live/src/config.test.ts | 179 ++ packages/qwen-live/src/config.ts | 183 ++ packages/qwen-live/src/daemon.test.ts | 357 +++ packages/qwen-live/src/daemon.ts | 336 +++ packages/qwen-live/src/host/discovery.test.ts | 346 +++ packages/qwen-live/src/host/discovery.ts | 618 +++++ .../src/host/live-host-coordinator.test.ts | 1206 +++++++++ .../src/host/live-host-coordinator.ts | 1377 +++++++++++ .../src/host/live-host-installer.test.ts | 367 +++ .../qwen-live/src/host/live-host-installer.ts | 504 ++++ packages/qwen-live/src/host/types.ts | 197 ++ packages/qwen-live/src/index.ts | 112 + .../qwen-live/src/log/session-log.test.ts | 163 ++ packages/qwen-live/src/log/session-log.ts | 148 ++ packages/qwen-live/src/logger.ts | 48 + .../src/orchestrator/injector.test.ts | 414 ++++ .../qwen-live/src/orchestrator/injector.ts | 260 ++ .../src/orchestrator/live-session.test.ts | 952 +++++++ .../src/orchestrator/live-session.ts | 1140 +++++++++ .../src/permissions/permission-broker.test.ts | 378 +++ .../src/permissions/permission-broker.ts | 240 ++ .../qwen-live/src/realtime/instructions.ts | 58 + .../src/realtime/realtime-session.test.ts | 1363 ++++++++++ .../src/realtime/realtime-session.ts | 2195 +++++++++++++++++ .../qwen-live/src/realtime/sanitize.test.ts | 29 + packages/qwen-live/src/realtime/sanitize.ts | 23 + packages/qwen-live/src/realtime/socket.ts | 37 + packages/qwen-live/src/tools/definitions.ts | 213 ++ .../qwen-live/src/tools/dispatcher.test.ts | 206 ++ packages/qwen-live/src/tools/dispatcher.ts | 124 + packages/qwen-live/src/tools/handles.test.ts | 210 ++ packages/qwen-live/src/tools/handles.ts | 161 ++ packages/qwen-live/tsconfig.build.json | 9 + packages/qwen-live/tsconfig.json | 11 + packages/qwen-live/vitest.config.ts | 29 + .../vscode-ide-companion/src/diff-manager.ts | 10 +- scripts/build.js | 1 + scripts/tests/no-ak-integration-ci.test.js | 4 + scripts/tests/package-scripts.test.js | 19 + scripts/version.js | 4 +- vitest.config.ts | 1 + 60 files changed, 18728 insertions(+), 9 deletions(-) create mode 100644 integration-tests/fake-dashscope-server.ts create mode 100644 integration-tests/qwen-live-harness.ts create mode 100644 integration-tests/qwen-live-m1-call.test.ts create mode 100644 integration-tests/qwen-live-m2-inject.test.ts create mode 100644 integration-tests/qwen-live-m2-permission.test.ts create mode 100644 integration-tests/qwen-live-m2-steering.test.ts create mode 100644 packages/qwen-live/.gitignore create mode 100644 packages/qwen-live/README.md create mode 100644 packages/qwen-live/build.mjs create mode 100644 packages/qwen-live/package.json create mode 100644 packages/qwen-live/src/adaptor/qwen-code-adaptor.test.ts create mode 100644 packages/qwen-live/src/adaptor/qwen-code-adaptor.ts create mode 100644 packages/qwen-live/src/adaptor/types.ts create mode 100644 packages/qwen-live/src/config.test.ts create mode 100644 packages/qwen-live/src/config.ts create mode 100644 packages/qwen-live/src/daemon.test.ts create mode 100644 packages/qwen-live/src/daemon.ts create mode 100644 packages/qwen-live/src/host/discovery.test.ts create mode 100644 packages/qwen-live/src/host/discovery.ts create mode 100644 packages/qwen-live/src/host/live-host-coordinator.test.ts create mode 100644 packages/qwen-live/src/host/live-host-coordinator.ts create mode 100644 packages/qwen-live/src/host/live-host-installer.test.ts create mode 100644 packages/qwen-live/src/host/live-host-installer.ts create mode 100644 packages/qwen-live/src/host/types.ts create mode 100644 packages/qwen-live/src/index.ts create mode 100644 packages/qwen-live/src/log/session-log.test.ts create mode 100644 packages/qwen-live/src/log/session-log.ts create mode 100644 packages/qwen-live/src/logger.ts create mode 100644 packages/qwen-live/src/orchestrator/injector.test.ts create mode 100644 packages/qwen-live/src/orchestrator/injector.ts create mode 100644 packages/qwen-live/src/orchestrator/live-session.test.ts create mode 100644 packages/qwen-live/src/orchestrator/live-session.ts create mode 100644 packages/qwen-live/src/permissions/permission-broker.test.ts create mode 100644 packages/qwen-live/src/permissions/permission-broker.ts create mode 100644 packages/qwen-live/src/realtime/instructions.ts create mode 100644 packages/qwen-live/src/realtime/realtime-session.test.ts create mode 100644 packages/qwen-live/src/realtime/realtime-session.ts create mode 100644 packages/qwen-live/src/realtime/sanitize.test.ts create mode 100644 packages/qwen-live/src/realtime/sanitize.ts create mode 100644 packages/qwen-live/src/realtime/socket.ts create mode 100644 packages/qwen-live/src/tools/definitions.ts create mode 100644 packages/qwen-live/src/tools/dispatcher.test.ts create mode 100644 packages/qwen-live/src/tools/dispatcher.ts create mode 100644 packages/qwen-live/src/tools/handles.test.ts create mode 100644 packages/qwen-live/src/tools/handles.ts create mode 100644 packages/qwen-live/tsconfig.build.json create mode 100644 packages/qwen-live/tsconfig.json create mode 100644 packages/qwen-live/vitest.config.ts diff --git a/.github/workflows/live-host.yml b/.github/workflows/live-host.yml index 4c188b42046..53aa98272fb 100644 --- a/.github/workflows/live-host.yml +++ b/.github/workflows/live-host.yml @@ -8,6 +8,10 @@ on: - 'packages/cli/src/serve/conversations/**' - 'packages/cli/src/serve/live/**' - 'packages/live-host/**' + # The protocol parity test couples the qwen-live host copies into + # this lane; without the path filter a qwen-live-only PR skips it + # and the first red lands in the merge queue instead. + - 'packages/qwen-live/src/host/**' - 'packages/sdk-typescript/src/daemon/types.ts' merge_group: workflow_dispatch: diff --git a/integration-tests/fake-dashscope-server.ts b/integration-tests/fake-dashscope-server.ts new file mode 100644 index 00000000000..4322bf1c6b2 --- /dev/null +++ b/integration-tests/fake-dashscope-server.ts @@ -0,0 +1,365 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Scriptable stand-in for the DashScope qwen-omni realtime endpoint that + * `packages/qwen-live` connects to (`ws://…/api-ws/v1/realtime?model=…`). + * + * Wire behavior mirrors what the daemon's realtime client + * (`packages/qwen-live/src/realtime/realtime-session.ts`) expects: + * - on connect the server sends `session.created`; the client answers with + * `session.update`, which we acknowledge with `session.updated` so the + * client resolves its open() promise; + * - every client message is recorded in `inbox` for assertions + * (session.update payload, input_audio_buffer.append frames, + * conversation.item.create context/function_call_output items, + * response.create, …); + * - the test drives the model side through the per-connection script API + * (`speakTranscript`, `functionCall`, `respondWithAudio`, + * `beginResponse`/`finishResponse`). Message shapes follow the provider + * events the client's `case` branches parse (see the responseCreated / + * functionCall / responseDone helpers in + * packages/qwen-live/src/realtime/realtime-session.test.ts). + * + * `autoAckResponses` (default on) answers client-initiated `response.create` + * requests (speakToUser flow) with an immediate `response.created` + + * `response.done` pair so the client's response arbitration never stalls. + * Tests that need a response held in flight open one manually with + * `beginResponse()` and settle it with `finishResponse()`. + */ + +import { EventEmitter } from 'node:events'; +import { createServer } from 'node:http'; +import type { AddressInfo } from 'node:net'; +import { WebSocketServer, type WebSocket } from 'ws'; + +type JsonObject = Record; + +function isRecord(value: unknown): value is JsonObject { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +export interface FakeDashScopeFunctionCall { + name: string; + /** Raw JSON string handed to the client as `item.arguments`. */ + argumentsJson: string; + callId: string; +} + +export interface FakeDashScopeConnection { + readonly index: number; + readonly socket: WebSocket; + /** Raw upgrade-request URL (path + query). */ + readonly requestUrl: string; + /** `model` query parameter from the upgrade URL. */ + readonly model: string | undefined; + /** `Authorization` header presented by the client. */ + readonly authorization: string | undefined; + /** Send one raw provider event (an `event_id` is added automatically). */ + send(message: JsonObject): void; + /** + * Simulate one finished user utterance: + * speech_started → speech_stopped → input_audio_buffer.committed → + * conversation.item.input_audio_transcription.completed. + * Returns the item id. + */ + speakTranscript(text: string): string; + /** + * Simulate the model calling a tool: response.created → + * response.output_item.done (item.type = 'function_call') → response.done. + * Returns the response id. + */ + functionCall(call: FakeDashScopeFunctionCall): string; + /** + * Simulate a direct spoken answer: response.created → + * response.audio.delta (base64) → response.audio.done → response.done. + * Returns the response id. + */ + respondWithAudio(pcm16: Uint8Array): string; + /** Open a response and leave it in flight. Returns the response id. */ + beginResponse(): string; + /** Settle a response previously opened with beginResponse(). */ + finishResponse(responseId: string, status?: string): void; +} + +export interface WaitForMessageOptions { + timeoutMs?: number; + /** Only consider inbox entries at or after this index (default 0). */ + fromIndex?: number; + description?: string; +} + +export interface FakeDashScopeServer { + /** HTTP origin, e.g. `http://127.0.0.1:PORT` — feed to the daemon as QWEN_LIVE_REALTIME_ENDPOINT. */ + url: string; + port: number; + connections: FakeDashScopeConnection[]; + /** Every JSON message received from every connection, in arrival order. */ + inbox: JsonObject[]; + /** Answer client `response.create` with created+done automatically. */ + autoAckResponses: boolean; + waitForConnection(timeoutMs?: number): Promise; + waitForMessage( + predicate: (message: JsonObject) => boolean, + opts?: WaitForMessageOptions, + ): Promise; + close(): Promise; +} + +/** + * If `message` is a `conversation.item.create` carrying a + * `function_call_output` item, return its call id and output string. + */ +export function functionCallOutputOf( + message: JsonObject, +): { callId: string; output: string } | undefined { + if (message['type'] !== 'conversation.item.create') return undefined; + const item = message['item']; + if (!isRecord(item) || item['type'] !== 'function_call_output') { + return undefined; + } + if ( + typeof item['call_id'] !== 'string' || + typeof item['output'] !== 'string' + ) { + return undefined; + } + return { callId: item['call_id'], output: item['output'] }; +} + +/** + * If `message` is a `conversation.item.create` carrying a user text item + * (the daemon's context/speech injections), return its text body. + */ +export function contextTextOf(message: JsonObject): string | undefined { + if (message['type'] !== 'conversation.item.create') return undefined; + const item = message['item']; + if (!isRecord(item) || item['type'] !== 'message') return undefined; + const content = item['content']; + if (!Array.isArray(content)) return undefined; + const first: unknown = content[0]; + if (!isRecord(first) || first['type'] !== 'input_text') return undefined; + return typeof first['text'] === 'string' ? first['text'] : undefined; +} + +export async function startFakeDashScopeServer(): Promise { + const httpServer = createServer((_req, res) => { + res.statusCode = 404; + res.end(); + }); + const wss = new WebSocketServer({ server: httpServer }); + const emitter = new EventEmitter(); + emitter.setMaxListeners(0); + + const connections: FakeDashScopeConnection[] = []; + const inbox: JsonObject[] = []; + let eventSeq = 0; + let itemSeq = 0; + let responseSeq = 0; + + const handle: FakeDashScopeServer = { + url: '', + port: 0, + connections, + inbox, + autoAckResponses: true, + waitForConnection: (timeoutMs = 15_000) => { + if (connections.length > 0) return Promise.resolve(connections[0]); + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { + emitter.off('connection', onConnection); + reject( + new Error( + `fake DashScope: no realtime connection within ${timeoutMs}ms`, + ), + ); + }, timeoutMs); + const onConnection = (connection: FakeDashScopeConnection) => { + clearTimeout(timer); + resolve(connection); + }; + emitter.once('connection', onConnection); + }); + }, + waitForMessage: (predicate, opts = {}) => { + const fromIndex = opts.fromIndex ?? 0; + const timeoutMs = opts.timeoutMs ?? 15_000; + for (let i = Math.max(fromIndex, 0); i < inbox.length; i++) { + if (predicate(inbox[i])) return Promise.resolve(inbox[i]); + } + return new Promise((resolve, reject) => { + const cleanup = () => { + clearTimeout(timer); + emitter.off('message', onMessage); + }; + const timer = setTimeout(() => { + cleanup(); + const recent = inbox + .slice(-15) + .map((message) => String(message['type'])) + .join(', '); + reject( + new Error( + `fake DashScope: timed out after ${timeoutMs}ms waiting for ` + + `${opts.description ?? 'a matching client message'} ` + + `(inbox=${inbox.length}, recent types: ${recent})`, + ), + ); + }, timeoutMs); + const onMessage = (message: JsonObject) => { + if (!predicate(message)) return; + cleanup(); + resolve(message); + }; + emitter.on('message', onMessage); + }); + }, + close: () => { + for (const connection of connections) { + try { + connection.socket.terminate(); + } catch { + /* already gone */ + } + } + wss.close(); + return new Promise((resolve) => { + httpServer.close(() => { + resolve(); + }); + }); + }, + }; + + wss.on('connection', (socket: WebSocket, req) => { + const requestUrl = req.url ?? ''; + const query = new URL(requestUrl, 'http://127.0.0.1').searchParams; + const authorization = + typeof req.headers['authorization'] === 'string' + ? req.headers['authorization'] + : undefined; + + const sendJson = (body: JsonObject) => { + if (socket.readyState !== socket.OPEN) return; + socket.send(JSON.stringify({ event_id: `evt-${++eventSeq}`, ...body })); + }; + + const beginResponse = (): string => { + const responseId = `resp-${++responseSeq}`; + sendJson({ + type: 'response.created', + response: { id: responseId, status: 'in_progress' }, + }); + return responseId; + }; + const finishResponse = (responseId: string, status = 'completed') => { + sendJson({ + type: 'response.done', + response: { id: responseId, status }, + }); + }; + + const connection: FakeDashScopeConnection = { + index: connections.length, + socket, + requestUrl, + model: query.get('model') ?? undefined, + authorization, + send: sendJson, + speakTranscript: (text) => { + const itemId = `item-${++itemSeq}`; + sendJson({ + type: 'input_audio_buffer.speech_started', + item_id: itemId, + audio_start_ms: 0, + }); + sendJson({ + type: 'input_audio_buffer.speech_stopped', + item_id: itemId, + audio_end_ms: 400, + }); + sendJson({ type: 'input_audio_buffer.committed', item_id: itemId }); + sendJson({ + type: 'conversation.item.input_audio_transcription.completed', + item_id: itemId, + transcript: text, + }); + return itemId; + }, + functionCall: (call) => { + const responseId = beginResponse(); + sendJson({ + type: 'response.output_item.done', + response_id: responseId, + item: { + id: `item-${call.callId}`, + type: 'function_call', + name: call.name, + call_id: call.callId, + arguments: call.argumentsJson, + }, + }); + finishResponse(responseId); + return responseId; + }, + respondWithAudio: (pcm16) => { + if (pcm16.byteLength === 0 || pcm16.byteLength % 2 !== 0) { + throw new Error('respondWithAudio needs a non-empty PCM16 buffer'); + } + const responseId = beginResponse(); + sendJson({ + type: 'response.audio.delta', + response_id: responseId, + item_id: `item-audio-${responseSeq}`, + delta: Buffer.from(pcm16).toString('base64'), + }); + sendJson({ type: 'response.audio.done', response_id: responseId }); + finishResponse(responseId); + return responseId; + }, + beginResponse, + finishResponse, + }; + connections.push(connection); + + socket.on('message', (data, isBinary) => { + if (isBinary) return; // the realtime protocol is JSON-text only + let parsed: unknown; + try { + parsed = JSON.parse(String(data)); + } catch { + return; + } + if (!isRecord(parsed)) return; + inbox.push(parsed); + if (parsed['type'] === 'session.update') { + sendJson({ type: 'session.updated', session: { id: 'sess-1' } }); + } else if ( + parsed['type'] === 'response.create' && + handle.autoAckResponses + ) { + finishResponse(beginResponse()); + } + emitter.emit('message', parsed); + }); + + // First provider event on a fresh connection; the client answers with + // session.update. + sendJson({ type: 'session.created' }); + emitter.emit('connection', connection); + }); + + await new Promise((resolve, reject) => { + httpServer.once('error', reject); + httpServer.listen(0, '127.0.0.1', () => { + resolve(); + }); + }); + const address = httpServer.address() as AddressInfo; + handle.port = address.port; + handle.url = `http://127.0.0.1:${address.port}`; + return handle; +} diff --git a/integration-tests/qwen-live-harness.ts b/integration-tests/qwen-live-harness.ts new file mode 100644 index 00000000000..e57930d5663 --- /dev/null +++ b/integration-tests/qwen-live-harness.ts @@ -0,0 +1,738 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * e2e harness for the standalone `qwen-live` daemon (packages/qwen-live). + * + * Everything here talks to real subprocesses over real network/file + * boundaries — no package sources are imported: + * - `spawnQwenLive` boots `node packages/qwen-live/dist/index.js` with a + * hermetic env (tmp data/discovery dirs, fake DashScope endpoint, a real + * `qwen serve` URL) and parses the single machine-readable stdout line, + * following the `_daemon-harness.spawnDaemon` pattern. + * - `FakeHost` speaks Host protocol v6 against the daemon's `/live/host` + * WebSocket: discovery-file lookup, Bearer token + `x-qwen-live-nonce` + * headers, `host.hello`, auto `host.pong`, auto success replies to + * `host.capture_screen_context`, `host.action`, binary input + * audio frames (8-byte BigUInt64BE epoch prefix + PCM16), and records + * `host.welcome`/`host.state` plus raw output PCM frames. + * - `bootLiveStack` assembles the full fixture: tmp workspace + HOME, a + * fake OpenAI model endpoint, a real `qwen serve`, a fake DashScope + * realtime server, the qwen-live daemon, and a connected FakeHost. + * - `waitForLiveLogEvents` reads the daemon's append-only session JSONL + * (`/sessions/*.jsonl`) as a synchronization point for + * daemon-internal happenings (e.g. "the backend turn_complete event + * reached the orchestrator") that have no other observable edge. + */ + +import { spawn, type ChildProcess } from 'node:child_process'; +import { randomUUID } from 'node:crypto'; +import { EventEmitter } from 'node:events'; +import { + mkdirSync, + mkdtempSync, + realpathSync, + rmSync, + writeFileSync, +} from 'node:fs'; +import { readFile, readdir } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import * as path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import WebSocket from 'ws'; +import { + sleep, + spawnDaemon, + type SpawnedDaemon, +} from './cli/_daemon-harness.js'; +import { + startFakeDashScopeServer, + type FakeDashScopeServer, +} from './fake-dashscope-server.js'; +import { + startFakeOpenAIServer, + type FakeOpenAIHandler, + type FakeOpenAIServer, +} from './fake-openai-server.js'; + +type JsonObject = Record; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const QWEN_LIVE_BIN = path.resolve( + __dirname, + '../packages/qwen-live/dist/index.js', +); + +export const QWEN_LIVE_REALTIME_MODEL = 'fake-omni-realtime'; +export const QWEN_LIVE_API_KEY = 'sk-test'; +const SERVE_TOKEN = 'qwen-live-e2e-token'; +const LIVE_LISTENING_RE = /qwen-live listening on http:\/\/127\.0\.0\.1:(\d+)/; +const DISPOSE_GRACE_MS = 10_000; +const LIVE_HOST_BUNDLE_ID = 'com.alibaba.qwen-code.live-host'; +const LIVE_HOST_PROTOCOL_VERSION = 6; +const LIVE_INPUT_AUDIO_EPOCH_BYTES = 8; + +// -- small async utilities ---------------------------------------------------- + +export interface Deferred { + promise: Promise; + resolve: (value: T) => void; + reject: (error: unknown) => void; +} + +export function deferred(): Deferred { + let resolve!: (value: T) => void; + let reject!: (error: unknown) => void; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + return { promise, resolve, reject }; +} + +export function withTimeout( + promise: Promise, + timeoutMs: number, + label: string, +): Promise { + let timer: ReturnType | undefined; + const timeout = new Promise((_, reject) => { + timer = setTimeout(() => { + reject(new Error(`timed out after ${timeoutMs}ms waiting for ${label}`)); + }, timeoutMs); + }); + return Promise.race([promise, timeout]).finally(() => { + if (timer !== undefined) clearTimeout(timer); + }) as Promise; +} + +// -- discovery --------------------------------------------------------------- + +export interface LiveDiscoveryRecordShape { + url: string; + token?: string; + protocolVersion: number; + pid: number; + instanceNonce: string; +} + +export function liveDiscoveryPath(discoveryDir: string): string { + return path.join(discoveryDir, 'live', 'daemon.json'); +} + +export async function readLiveDiscovery( + discoveryDir: string, +): Promise { + const raw = await readFile(liveDiscoveryPath(discoveryDir), 'utf8'); + return JSON.parse(raw) as LiveDiscoveryRecordShape; +} + +// -- qwen-live daemon process -------------------------------------------------- + +export interface SpawnQwenLiveOptions { + serveUrl: string; + serveToken: string; + realtimeEndpoint: string; + dataDir: string; + discoveryDir: string; + /** Default workspace cwd for handoff-created sessions (QWEN_LIVE_CWD). */ + cwd: string; + apiKey?: string; + model?: string; + bootTimeoutMs?: number; + env?: Record; +} + +export interface SpawnedQwenLive { + proc: ChildProcess; + port: number; + url: string; + stdoutBuf: { value: string }; + stderrBuf: { value: string }; + /** Idempotent. SIGTERM, wait up to 10s, then SIGKILL. */ + dispose: () => Promise; +} + +export async function spawnQwenLive( + opts: SpawnQwenLiveOptions, +): Promise { + const bootTimeoutMs = opts.bootTimeoutMs ?? 15_000; + const proc = spawn(process.execPath, [QWEN_LIVE_BIN], { + stdio: ['ignore', 'pipe', 'pipe'], + env: { + ...process.env, + DASHSCOPE_API_KEY: opts.apiKey ?? QWEN_LIVE_API_KEY, + QWEN_LIVE_REALTIME_ENDPOINT: opts.realtimeEndpoint, + QWEN_LIVE_REALTIME_MODEL: opts.model ?? QWEN_LIVE_REALTIME_MODEL, + QWEN_LIVE_SERVE_URL: opts.serveUrl, + QWEN_SERVER_TOKEN: opts.serveToken, + QWEN_LIVE_DATA_DIR: opts.dataDir, + QWEN_LIVE_DISCOVERY_DIR: opts.discoveryDir, + QWEN_LIVE_CWD: opts.cwd, + QWEN_LIVE_PORT: '0', + ...opts.env, + }, + }); + + const stdoutBuf = { value: '' }; + const stderrBuf = { value: '' }; + proc.stdout?.on('data', (chunk: Buffer) => { + stdoutBuf.value += chunk.toString(); + }); + proc.stderr?.on('data', (chunk: Buffer) => { + stderrBuf.value += chunk.toString(); + }); + + const port = await new Promise((resolve, reject) => { + let settled = false; + const cleanup = () => { + proc.stdout?.off('data', onData); + proc.off('exit', onExit); + clearTimeout(bootTimer); + }; + const fail = (error: Error, kill = false) => { + if (settled) return; + settled = true; + cleanup(); + if (kill && proc.exitCode === null) proc.kill('SIGTERM'); + reject(error); + }; + const bootTimer = setTimeout(() => { + fail( + new Error( + `qwen-live boot timeout after ${bootTimeoutMs}ms:\n` + + `stdout=${stdoutBuf.value}\nstderr=${stderrBuf.value}`, + ), + true, + ); + }, bootTimeoutMs); + const onData = () => { + const match = stdoutBuf.value.match(LIVE_LISTENING_RE); + if (!match || settled) return; + settled = true; + cleanup(); + resolve(Number(match[1])); + }; + const onExit = (code: number | null) => { + fail( + new Error( + `qwen-live exited with ${code} before listening:\n` + + `stdout=${stdoutBuf.value}\nstderr=${stderrBuf.value}`, + ), + ); + }; + proc.stdout!.on('data', onData); + proc.once('exit', onExit); + }); + + const dispose = async () => { + // Signal death leaves exitCode null and sets signalCode instead — both + // mean the child is already gone and the 'exit' event already fired. + if (proc.exitCode !== null || proc.signalCode !== null) return; + proc.kill('SIGTERM'); + await new Promise((resolve) => { + const timer = setTimeout(() => { + try { + proc.kill('SIGKILL'); + } catch { + /* already gone */ + } + resolve(); + }, DISPOSE_GRACE_MS); + proc.once('exit', () => { + clearTimeout(timer); + resolve(); + }); + }); + }; + + return { + proc, + port, + url: `http://127.0.0.1:${port}`, + stdoutBuf, + stderrBuf, + dispose, + }; +} + +// -- fake Host (protocol v6) --------------------------------------------------- + +export interface FakeHostStateEntry { + type: 'host.welcome' | 'host.state'; + epoch: number; + status: JsonObject; +} + +export interface FakeHostWaitOptions { + timeoutMs?: number; + /** Only consider entries at or after this index (default 0). */ + fromIndex?: number; +} + +export class FakeHost { + /** Every parsed text frame from the daemon, in arrival order. */ + readonly messages: JsonObject[] = []; + /** host.welcome / host.state frames, normalized. */ + readonly states: FakeHostStateEntry[] = []; + /** Raw binary output frames (bare PCM16). */ + readonly audioFrames: Buffer[] = []; + + private socket: WebSocket | undefined; + private readonly emitter = new EventEmitter(); + private readonly hostInstanceNonce = randomUUID(); + private screenshotPath: string | undefined; + + constructor(private readonly discoveryDir: string) { + this.emitter.setMaxListeners(0); + } + + /** Read the discovery file, connect, send host.hello, await host.welcome. */ + async connect(timeoutMs = 10_000): Promise { + const record = await readLiveDiscovery(this.discoveryDir); + const socket = new WebSocket( + `${record.url.replace(/^http/, 'ws')}/live/host`, + { + headers: { + Authorization: `Bearer ${record.token ?? ''}`, + 'x-qwen-live-nonce': record.instanceNonce, + }, + }, + ); + this.socket = socket; + socket.on('message', (data, isBinary) => { + if (isBinary) { + const frame = Buffer.isBuffer(data) + ? Buffer.from(data) + : Array.isArray(data) + ? Buffer.concat(data) + : Buffer.from(data as ArrayBuffer); + this.audioFrames.push(frame); + this.emitter.emit('audio', frame); + return; + } + let parsed: unknown; + try { + parsed = JSON.parse(String(data)); + } catch { + return; + } + if (typeof parsed !== 'object' || parsed === null) return; + const message = parsed as JsonObject; + this.messages.push(message); + if (message['type'] === 'host.ping') { + this.send({ type: 'host.pong', pingId: message['pingId'] }); + } else if (message['type'] === 'host.capture_screen_context') { + // The hello advertises `appshot: true`, so the daemon may request a + // capture; settle it immediately (a real Host replies with a + // screenshot it wrote to disk plus the accessibility dump). + this.send({ + type: 'host.screen_context_result', + requestId: message['requestId'], + success: true, + appName: 'FakeApp', + windowTitle: 'Fake Window', + accessibilityText: 'fake accessibility text', + screenshotPath: this.fakeScreenshotPath(), + }); + } else if ( + message['type'] === 'host.welcome' || + message['type'] === 'host.state' + ) { + const entry: FakeHostStateEntry = { + type: message['type'], + epoch: Number(message['epoch']), + status: (message['status'] ?? {}) as JsonObject, + }; + this.states.push(entry); + this.emitter.emit('state', entry); + } + this.emitter.emit('message', message); + }); + socket.on('close', (code, reason) => { + this.emitter.emit('close', code, String(reason)); + }); + + await withTimeout( + new Promise((resolve, reject) => { + socket.once('open', () => { + resolve(); + }); + socket.once('error', reject); + }), + timeoutMs, + 'the /live/host WebSocket to open', + ); + this.send({ + type: 'host.hello', + protocolVersion: LIVE_HOST_PROTOCOL_VERSION, + hostVersion: '1.0.0-e2e', + bundleId: LIVE_HOST_BUNDLE_ID, + instanceNonce: this.hostInstanceNonce, + permissions: { + microphone: 'granted', + accessibility: 'granted', + screenRecording: 'granted', + }, + selfChecks: { + audioInput: true, + audioOutput: true, + globalShortcut: true, + appshot: true, + }, + }); + await this.waitForState((entry) => entry.type === 'host.welcome', { + timeoutMs, + }); + } + + action(action: 'toggle' | 'new' | 'stop'): void { + this.send({ type: 'host.action', action }); + } + + /** Input audio frame: 8-byte BigUInt64BE epoch prefix + PCM16 payload. */ + sendAudio(epoch: number, pcm16: Buffer): void { + const frame = Buffer.alloc(LIVE_INPUT_AUDIO_EPOCH_BYTES + pcm16.byteLength); + frame.writeBigUInt64BE(BigInt(epoch), 0); + pcm16.copy(frame, LIVE_INPUT_AUDIO_EPOCH_BYTES); + this.socketOrThrow().send(frame, { binary: true }); + } + + waitForState( + predicate: (entry: FakeHostStateEntry) => boolean, + opts: FakeHostWaitOptions = {}, + ): Promise { + const fromIndex = opts.fromIndex ?? 0; + const timeoutMs = opts.timeoutMs ?? 15_000; + for (let i = Math.max(fromIndex, 0); i < this.states.length; i++) { + if (predicate(this.states[i])) return Promise.resolve(this.states[i]); + } + return new Promise((resolve, reject) => { + const cleanup = () => { + clearTimeout(timer); + this.emitter.off('state', onState); + }; + const timer = setTimeout(() => { + cleanup(); + const recent = this.states + .slice(-8) + .map((entry) => `${entry.type}:${String(entry.status['state'])}`) + .join(', '); + reject( + new Error( + `FakeHost: timed out after ${timeoutMs}ms waiting for a host ` + + `state (recent: ${recent})`, + ), + ); + }, timeoutMs); + const onState = (entry: FakeHostStateEntry) => { + if (!predicate(entry)) return; + cleanup(); + resolve(entry); + }; + this.emitter.on('state', onState); + }); + } + + waitForAudioFrame(opts: FakeHostWaitOptions = {}): Promise { + const fromIndex = opts.fromIndex ?? 0; + const timeoutMs = opts.timeoutMs ?? 15_000; + if (this.audioFrames.length > fromIndex) { + return Promise.resolve(this.audioFrames[fromIndex]); + } + return new Promise((resolve, reject) => { + const cleanup = () => { + clearTimeout(timer); + this.emitter.off('audio', onAudio); + }; + const timer = setTimeout(() => { + cleanup(); + reject( + new Error( + `FakeHost: timed out after ${timeoutMs}ms waiting for an output ` + + `audio frame (have ${this.audioFrames.length})`, + ), + ); + }, timeoutMs); + const onAudio = () => { + if (this.audioFrames.length <= fromIndex) return; + cleanup(); + resolve(this.audioFrames[fromIndex]); + }; + this.emitter.on('audio', onAudio); + }); + } + + close(): void { + try { + this.socket?.close(); + } catch { + /* already closed */ + } + } + + private send(message: JsonObject): void { + this.socketOrThrow().send(JSON.stringify(message)); + } + + /** + * Lazily materialize a real (1x1) PNG for `host.screen_context_result`: + * the daemon registers the path as an asset, so it should exist on disk. + * Written into the discovery dir, which the fixture already tears down. + */ + private fakeScreenshotPath(): string { + if (!this.screenshotPath) { + const file = path.join(this.discoveryDir, 'fake-appshot.png'); + writeFileSync( + file, + Buffer.from( + 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8' + + 'z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==', + 'base64', + ), + ); + this.screenshotPath = file; + } + return this.screenshotPath; + } + + private socketOrThrow(): WebSocket { + if (!this.socket) throw new Error('FakeHost is not connected'); + return this.socket; + } +} + +// -- daemon session log (JSONL) ------------------------------------------------ + +export interface LiveLogEvent { + ts: number; + seq: number; + type: string; + payload: JsonObject; +} + +export async function readLiveLogEvents( + dataDir: string, +): Promise { + const sessionsDir = path.join(dataDir, 'sessions'); + let files: string[]; + try { + files = await readdir(sessionsDir); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return []; + throw error; + } + const events: LiveLogEvent[] = []; + for (const file of files.sort()) { + if (!file.includes('.jsonl')) continue; + let raw: string; + try { + raw = await readFile(path.join(sessionsDir, file), 'utf8'); + } catch { + continue; + } + for (const line of raw.split('\n')) { + if (!line.trim()) continue; + try { + events.push(JSON.parse(line) as LiveLogEvent); + } catch { + /* torn tail line while the daemon is mid-write */ + } + } + } + return events; +} + +/** + * Wait until at least `minCount` session-log events match `predicate`. + * The JSONL log has no push channel, so this is a bounded condition-poll + * with an explicit deadline (25ms period). + */ +export async function waitForLiveLogEvents( + dataDir: string, + predicate: (event: LiveLogEvent) => boolean, + opts: { minCount?: number; timeoutMs?: number; description?: string } = {}, +): Promise { + const minCount = opts.minCount ?? 1; + const timeoutMs = opts.timeoutMs ?? 30_000; + const deadline = Date.now() + timeoutMs; + for (;;) { + const matches = (await readLiveLogEvents(dataDir)).filter(predicate); + if (matches.length >= minCount) return matches; + if (Date.now() > deadline) { + throw new Error( + `timed out after ${timeoutMs}ms waiting for ` + + `${opts.description ?? 'live session-log events'} ` + + `(${matches.length}/${minCount})`, + ); + } + await sleep(25); + } +} + +// -- full fixture --------------------------------------------------------------- + +export interface BootLiveStackOptions { + /** + * Builds the fake OpenAI handler once the fixture directories exist, so + * handlers can reference paths inside the workspace. + */ + makeOpenAIHandler: (info: { workspaceDir: string }) => FakeOpenAIHandler; +} + +export interface LiveStack { + workspaceDir: string; + homeDir: string; + dataDir: string; + discoveryDir: string; + fakeOpenAI: FakeOpenAIServer; + serve: SpawnedDaemon; + fakeDash: FakeDashScopeServer; + live: SpawnedQwenLive; + host: FakeHost; + /** + * Session created directly by the test harness to pre-spawn the serve + * daemon's ACP child (kept open for the fixture's lifetime). Tests that + * enumerate serve sessions should exclude this id. + */ + prewarmSessionId: string; + dispose(): Promise; +} + +export async function bootLiveStack( + options: BootLiveStackOptions, +): Promise { + const workspaceDir = realpathSync( + mkdtempSync(path.join(tmpdir(), 'qwen-live-e2e-ws-')), + ); + const homeDir = mkdtempSync(path.join(tmpdir(), 'qwen-live-e2e-home-')); + const dataDir = mkdtempSync(path.join(tmpdir(), 'qwen-live-e2e-data-')); + const discoveryDir = mkdtempSync(path.join(tmpdir(), 'qwen-live-e2e-disc-')); + const qwenHome = path.join(homeDir, '.qwen'); + mkdirSync(qwenHome, { recursive: true }); + // Keep the model round-trips predictable: no follow-up suggestion turns. + writeFileSync( + path.join(qwenHome, 'settings.json'), + JSON.stringify({ ui: { enableFollowupSuggestions: false } }), + ); + + const disposers: Array<() => Promise | void> = []; + const disposeAll = async () => { + for (const dispose of disposers.reverse()) { + try { + await dispose(); + } catch { + /* keep tearing down */ + } + } + // Debug escape hatch: keep the temp dirs (session JSONL logs live in + // `/sessions`) for post-mortem inspection. KEEP_OUTPUT=true is + // the repo-wide convention (test-helper.ts, globalSetup.ts) and is set + // by the e2e lanes; QWEN_LIVE_E2E_KEEP keeps only these fixtures. + if ( + !process.env['QWEN_LIVE_E2E_KEEP'] && + process.env['KEEP_OUTPUT'] !== 'true' + ) { + for (const dir of [workspaceDir, homeDir, dataDir, discoveryDir]) { + rmSync(dir, { recursive: true, force: true }); + } + } + }; + + try { + const fakeOpenAI = await startFakeOpenAIServer( + options.makeOpenAIHandler({ workspaceDir }), + ); + disposers.push(() => fakeOpenAI.close()); + + const serve = await spawnDaemon({ + workspaceCwd: workspaceDir, + token: SERVE_TOKEN, + bootTimeoutMs: 30_000, + env: { + HOME: homeDir, + QWEN_HOME: qwenHome, + QWEN_ACP_LOCAL_READ_ROOTS: '', + OPENAI_API_KEY: 'fake-key', + OPENAI_BASE_URL: fakeOpenAI.baseUrl, + OPENAI_MODEL: 'fake-model', + QWEN_MODEL: 'fake-model', + NO_PROXY: '127.0.0.1,localhost', + no_proxy: '127.0.0.1,localhost', + http_proxy: '', + https_proxy: '', + HTTP_PROXY: '', + HTTPS_PROXY: '', + ALL_PROXY: '', + all_proxy: '', + }, + }); + disposers.push(() => serve.dispose()); + + // Pre-spawn the serve daemon's ACP child so the qwen-live orchestrator's + // first createSession (inside the 5s tool-dispatch budget) is fast. + const prewarm = await serve.client.createOrAttachSession({ + workspaceCwd: workspaceDir, + sessionScope: 'thread', + }); + + const fakeDash = await startFakeDashScopeServer(); + disposers.push(() => fakeDash.close()); + + const live = await spawnQwenLive({ + serveUrl: serve.base, + serveToken: SERVE_TOKEN, + realtimeEndpoint: fakeDash.url, + dataDir, + discoveryDir, + cwd: workspaceDir, + }); + disposers.push(() => live.dispose()); + + const host = new FakeHost(discoveryDir); + disposers.push(() => { + host.close(); + }); + await host.connect(); + + return { + workspaceDir, + homeDir, + dataDir, + discoveryDir, + fakeOpenAI, + serve, + fakeDash, + live, + host, + prewarmSessionId: prewarm.sessionId, + dispose: disposeAll, + }; + } catch (error) { + await disposeAll(); + throw error; + } +} + +/** + * Drive the Host `toggle` action and wait for the call to become live: + * returns the realtime connection the daemon opened against the fake + * DashScope server and the call epoch (from the `listening` host.state). + */ +export async function startLiveCall(stack: LiveStack): Promise<{ + epoch: number; + conn: Awaited>; +}> { + const stateIndex = stack.host.states.length; + const connection = stack.fakeDash.waitForConnection(20_000); + stack.host.action('toggle'); + const conn = await connection; + const listening = await stack.host.waitForState( + (entry) => entry.status['state'] === 'listening', + { timeoutMs: 20_000, fromIndex: stateIndex }, + ); + return { epoch: listening.epoch, conn }; +} diff --git a/integration-tests/qwen-live-m1-call.test.ts b/integration-tests/qwen-live-m1-call.test.ts new file mode 100644 index 00000000000..564b9c9c9c3 --- /dev/null +++ b/integration-tests/qwen-live-m1-call.test.ts @@ -0,0 +1,249 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * qwen-live M1 — one full call, end to end, against real subprocesses: + * a real `qwen serve` (model side backed by the fake OpenAI server), the + * real `qwen-live` daemon binary, a fake DashScope realtime endpoint, and + * a protocol-v6 FakeHost. + * + * a. the discovery file exists with the documented fields; + * b. FakeHost connect → hello → host.welcome; + * c. `toggle` opens the realtime connection (auth header + model query), + * sends session.update with the 8-tool surface, and the call reaches + * `listening`; + * d. direct-answer path: Host input audio frames reach the provider as + * input_audio_buffer.append, provider output audio reaches the Host as + * bare PCM frames; + * e. handoff path: a handoff function call lands on the real serve daemon + * as a prompt; the receipt (function_call_output), the [COMPLETE] + * context injection, and the [SPEAK_TO_USER] + response.create speech + * request all arrive on the provider socket in order; + * f. SIGTERM removes the discovery file. + */ + +import { existsSync } from 'node:fs'; +import { readFile } from 'node:fs/promises'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { + contextTextOf, + functionCallOutputOf, + type FakeDashScopeConnection, +} from './fake-dashscope-server.js'; +import { + bootLiveStack, + liveDiscoveryPath, + QWEN_LIVE_API_KEY, + QWEN_LIVE_REALTIME_MODEL, + type LiveStack, +} from './qwen-live-harness.js'; + +// Windows: the harness relies on POSIX process semantics (SIGTERM shutdown +// path is part of scenario f). Container sandboxes: the serve daemon's ACP +// child cannot reach the host-loopback fake OpenAI server (same skip as +// qwen-serve-streaming.test.ts). +const SKIP = + process.platform === 'win32' || + Boolean( + process.env['QWEN_SANDBOX'] && + process.env['QWEN_SANDBOX'].toLowerCase() !== 'false', + ); +const describeE2E = SKIP ? describe.skip : describe; + +const EXPECTED_TOOL_NAMES = [ + 'appshot', + 'handoff', + 'remain_silent', + 'respond_permission', + 'session_create', + 'session_list', + 'session_monitor', + 'session_stop', +]; + +describeE2E('qwen-live M1 — end-to-end voice call', () => { + let stack: LiveStack; + let conn: FakeDashScopeConnection; + let callEpoch = 0; + + beforeAll(async () => { + stack = await bootLiveStack({ + makeOpenAIHandler: () => () => ({ + content: 'm1 backend turn complete', + }), + }); + }, 180_000); + + afterAll(async () => { + await stack?.dispose(); + }, 60_000); + + it('publishes a discovery record with the documented fields', async () => { + const record = JSON.parse( + await readFile(liveDiscoveryPath(stack.discoveryDir), 'utf8'), + ) as Record; + expect(record['url']).toBe(stack.live.url); + expect(typeof record['token']).toBe('string'); + expect(String(record['token']).length).toBeGreaterThan(0); + expect(record['protocolVersion']).toBe(6); + expect(record['pid']).toBe(stack.live.proc.pid); + expect(String(record['instanceNonce'])).toMatch(/^[A-Za-z0-9_-]{16,256}$/); + }); + + it('answers host.hello with host.welcome', () => { + const welcome = stack.host.states.find( + (entry) => entry.type === 'host.welcome', + ); + expect(welcome).toBeDefined(); + expect(Number.isInteger(welcome!.epoch)).toBe(true); + expect(welcome!.status['available']).toBe(true); + expect(welcome!.status['state']).toBe('idle'); + }); + + it('toggle connects to the realtime provider and reaches listening', async () => { + const stateIndex = stack.host.states.length; + const connection = stack.fakeDash.waitForConnection(20_000); + stack.host.action('toggle'); + conn = await connection; + + expect(conn.authorization).toBe(`Bearer ${QWEN_LIVE_API_KEY}`); + expect(conn.model).toBe(QWEN_LIVE_REALTIME_MODEL); + expect(conn.requestUrl).toContain('/api-ws/v1/realtime'); + + const update = await stack.fakeDash.waitForMessage( + (message) => message['type'] === 'session.update', + { timeoutMs: 10_000, description: 'session.update' }, + ); + const session = update['session'] as Record; + const tools = session['tools'] as Array<{ + type: string; + function: { name: string }; + }>; + expect(tools.map((tool) => tool.function.name).sort()).toEqual( + EXPECTED_TOOL_NAMES, + ); + expect(typeof session['instructions']).toBe('string'); + expect(String(session['instructions']).length).toBeGreaterThan(0); + + const listening = await stack.host.waitForState( + (entry) => entry.status['state'] === 'listening', + { timeoutMs: 20_000, fromIndex: stateIndex }, + ); + callEpoch = listening.epoch; + expect(callEpoch).toBeGreaterThan(0); + }); + + it('routes Host input audio up and provider output audio down', async () => { + const pcmIn = Buffer.alloc(3_200); + for (let i = 0; i < pcmIn.length; i++) pcmIn[i] = i % 251; + const inboxIndex = stack.fakeDash.inbox.length; + stack.host.sendAudio(callEpoch, pcmIn); + const append = await stack.fakeDash.waitForMessage( + (message) => message['type'] === 'input_audio_buffer.append', + { fromIndex: inboxIndex, description: 'input_audio_buffer.append' }, + ); + expect(Buffer.from(String(append['audio']), 'base64').equals(pcmIn)).toBe( + true, + ); + + const pcmOut = Buffer.alloc(4_800); + for (let i = 0; i < pcmOut.length; i++) pcmOut[i] = (i * 7) % 253; + const framesBefore = stack.host.audioFrames.length; + conn.respondWithAudio(pcmOut); + const frame = await stack.host.waitForAudioFrame({ + fromIndex: framesBefore, + }); + expect(frame.equals(pcmOut)).toBe(true); + }); + + it('hands off to the real serve daemon and injects the result back', async () => { + const inboxIndex = stack.fakeDash.inbox.length; + conn.speakTranscript('fix the failing test'); + conn.functionCall({ + name: 'handoff', + argumentsJson: '{"task":"fix the failing test"}', + callId: 'call-1', + }); + + // Receipt: the handoff was admitted by qwen serve. + const receiptMessage = await stack.fakeDash.waitForMessage( + (message) => functionCallOutputOf(message)?.callId === 'call-1', + { + timeoutMs: 30_000, + fromIndex: inboxIndex, + description: 'the handoff function_call_output receipt', + }, + ); + const receipt = JSON.parse( + functionCallOutputOf(receiptMessage)!.output, + ) as Record; + expect(receipt['status']).toBe('accepted'); + expect(receipt['job']).toBe('job_1'); + + // The real serve daemon received the prompt (model side = fake OpenAI). + await expect + .poll( + () => + stack.fakeOpenAI.requests.some((request) => + JSON.stringify(request.body['messages'] ?? []).includes( + 'fix the failing test', + ), + ), + { timeout: 30_000 }, + ) + .toBe(true); + + // The turn's conclusion flows back as a silent [COMPLETE] context item… + const completeMessage = await stack.fakeDash.waitForMessage( + (message) => { + const text = contextTextOf(message); + return text !== undefined && text.includes('[COMPLETE job_1]'); + }, + { + timeoutMs: 30_000, + fromIndex: inboxIndex, + description: 'the [COMPLETE job_1] context injection', + }, + ); + const completeText = contextTextOf(completeMessage)!; + expect(completeText).toMatch(/^\[BACKEND\] /); + expect(completeText).toContain('m1 backend turn complete'); + expect(stack.fakeDash.inbox.indexOf(receiptMessage)).toBeLessThan( + stack.fakeDash.inbox.indexOf(completeMessage), + ); + + // …and as a spoken line: a [SPEAK_TO_USER] item followed by + // response.create (speakToUser wire shape in realtime-session.ts). + const speakMessage = await stack.fakeDash.waitForMessage( + (message) => { + const text = contextTextOf(message); + return text !== undefined && text.startsWith('[SPEAK_TO_USER] '); + }, + { + timeoutMs: 15_000, + fromIndex: inboxIndex, + description: 'the [SPEAK_TO_USER] speech request item', + }, + ); + expect(contextTextOf(speakMessage)).toContain('job_1'); + const speakIndex = stack.fakeDash.inbox.indexOf(speakMessage); + await stack.fakeDash.waitForMessage( + (message) => message['type'] === 'response.create', + { + timeoutMs: 15_000, + fromIndex: speakIndex, + description: 'the response.create following [SPEAK_TO_USER]', + }, + ); + }); + + it('removes the discovery file on SIGTERM', async () => { + const discoveryFile = liveDiscoveryPath(stack.discoveryDir); + expect(existsSync(discoveryFile)).toBe(true); + await stack.live.dispose(); // SIGTERM → graceful stop + expect(existsSync(discoveryFile)).toBe(false); + }); +}); diff --git a/integration-tests/qwen-live-m2-inject.test.ts b/integration-tests/qwen-live-m2-inject.test.ts new file mode 100644 index 00000000000..2f276222c3b --- /dev/null +++ b/integration-tests/qwen-live-m2-inject.test.ts @@ -0,0 +1,234 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * qwen-live M2 — result-injection window discipline (orchestrator/injector): + * + * 1. a backend turn_complete that arrives while a realtime response is in + * flight is NOT injected; it is delivered right after the response + * settles (`response.done`); + * 2. several jobs finishing while the window is closed are batched into a + * single conversation.item.create context injection (Injector.flush + * joins the whole batch into one silent context message). + * + * The fake OpenAI handler gates each backend turn on a marker-keyed deferred + * so the test controls exactly when qwen serve finishes each turn. The + * daemon-side "turn_complete reached the orchestrator" edge is observed + * through the daemon's session JSONL log. + */ + +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { sleep } from './cli/_daemon-harness.js'; +import { + contextTextOf, + functionCallOutputOf, + type FakeDashScopeConnection, +} from './fake-dashscope-server.js'; +import { + bootLiveStack, + deferred, + startLiveCall, + waitForLiveLogEvents, + type Deferred, + type LiveStack, +} from './qwen-live-harness.js'; + +const SKIP = + process.platform === 'win32' || + Boolean( + process.env['QWEN_SANDBOX'] && + process.env['QWEN_SANDBOX'].toLowerCase() !== 'false', + ); +const describeE2E = SKIP ? describe.skip : describe; + +const isTurnComplete = (event: { + type: string; + payload: Record; +}) => + event.type === 'backend.event' && event.payload['type'] === 'turn_complete'; + +describeE2E('qwen-live M2 — injection window', () => { + /** Consumed by the fake model: a gate is deleted the moment it matches. */ + const gates = new Map(); + /** Stable test-side resolve handles (the fake model never touches these). */ + const gateHandles = new Map(); + let stack: LiveStack; + let conn: FakeDashScopeConnection; + let callSeq = 0; + + const gatedHandoff = async ( + task: string, + extraArgs: Record = {}, + ): Promise> => { + const gate = deferred(); + gates.set(task, gate); + gateHandles.set(task, gate); + const callId = `call-h${++callSeq}`; + conn.functionCall({ + name: 'handoff', + argumentsJson: JSON.stringify({ task, ...extraArgs }), + callId, + }); + const receiptMessage = await stack.fakeDash.waitForMessage( + (message) => functionCallOutputOf(message)?.callId === callId, + { timeoutMs: 30_000, description: `handoff receipt for ${task}` }, + ); + const receipt = JSON.parse( + functionCallOutputOf(receiptMessage)!.output, + ) as Record; + expect(receipt['status']).toBe('accepted'); + expect(typeof receipt['job']).toBe('string'); + return receipt; + }; + + /** Open a held response and wait until the daemon has registered it. */ + const holdResponse = async (): Promise => { + const stateIndex = stack.host.states.length; + const holdId = conn.beginResponse(); + await stack.host.waitForState( + (entry) => entry.status['state'] === 'speaking', + { timeoutMs: 10_000, fromIndex: stateIndex }, + ); + return holdId; + }; + + beforeAll(async () => { + stack = await bootLiveStack({ + makeOpenAIHandler: + () => + async ({ body }) => { + // qwen serve sends the WHOLE conversation history with every model + // request, and both tests share one backend session — so match + // markers only against the current turn's LAST message (earlier + // turns' markers stay in the history forever) and consume the gate + // on match so a settled gate can never vacuously match a later + // turn. + const messages = (body['messages'] ?? []) as unknown[]; + const lastMessage = JSON.stringify(messages.at(-1) ?? ''); + for (const [marker, gate] of gates) { + if (lastMessage.includes(marker)) { + gates.delete(marker); + await gate.promise; + return { content: `finished ${marker}` }; + } + } + return { content: 'ok' }; + }, + }); + conn = (await startLiveCall(stack)).conn; + }, 180_000); + + afterAll(async () => { + for (const gate of gateHandles.values()) gate.resolve(); // never leave serve hung + await stack?.dispose(); + }, 60_000); + + it('holds [COMPLETE] while a response is in flight and injects after response.done', async () => { + const inboxIndex = stack.fakeDash.inbox.length; + const receipt = await gatedHandoff('inject-window-task'); + const job = String(receipt['job']); + + // Close the injection window: a response is now in flight. + const holdId = await holdResponse(); + + // Let the backend turn finish and wait until the daemon's orchestrator + // has consumed the turn_complete event (session-log sync point). + gateHandles.get('inject-window-task')!.resolve(); + await waitForLiveLogEvents(stack.dataDir, isTurnComplete, { + minCount: 1, + timeoutMs: 30_000, + description: 'backend.event turn_complete (inject-window-task)', + }); + // Negative assertion needs a bounded settle window: a (buggy) premature + // injection would be written to this loopback socket within milliseconds + // of the log line above — there is no further event to await when the + // implementation is correct. + await sleep(250); + const premature = stack.fakeDash.inbox + .slice(inboxIndex) + .map((message) => contextTextOf(message) ?? '') + .filter((text) => text.includes('[COMPLETE')); + expect(premature).toEqual([]); + + // Reopen the window: the queued conclusion must now arrive. + conn.finishResponse(holdId); + const complete = await stack.fakeDash.waitForMessage( + (message) => { + const text = contextTextOf(message); + return text !== undefined && text.includes(`[COMPLETE ${job}]`); + }, + { + timeoutMs: 15_000, + fromIndex: inboxIndex, + description: `[COMPLETE ${job}] after response.done`, + }, + ); + expect(contextTextOf(complete)).toContain('finished inject-window-task'); + }); + + it('batches multiple completions into one context injection', async () => { + // A second backend session so two independent turns can complete. + conn.functionCall({ + name: 'session_create', + argumentsJson: JSON.stringify({ label: 'second workstream' }), + callId: 'call-sc', + }); + const createdMessage = await stack.fakeDash.waitForMessage( + (message) => functionCallOutputOf(message)?.callId === 'call-sc', + { timeoutMs: 30_000, description: 'session_create receipt' }, + ); + const created = JSON.parse( + functionCallOutputOf(createdMessage)!.output, + ) as Record; + expect(created['status']).toBe('ok'); + const secondSession = String(created['handle']); + + const receiptA = await gatedHandoff('batch-task-a'); + const receiptB = await gatedHandoff('batch-task-b', { + session: secondSession, + }); + expect(receiptB['session']).toBe(secondSession); + const jobA = String(receiptA['job']); + const jobB = String(receiptB['job']); + expect(jobA).not.toBe(jobB); + + const inboxIndex = stack.fakeDash.inbox.length; + const holdId = await holdResponse(); + + gateHandles.get('batch-task-a')!.resolve(); + gateHandles.get('batch-task-b')!.resolve(); + // 1 turn_complete from the previous test + 2 here. + await waitForLiveLogEvents(stack.dataDir, isTurnComplete, { + minCount: 3, + timeoutMs: 30_000, + description: 'backend.event turn_complete (batch-task-a/b)', + }); + await sleep(250); // bounded settle window for the negative assertion + const premature = stack.fakeDash.inbox + .slice(inboxIndex) + .map((message) => contextTextOf(message) ?? '') + .filter((text) => text.includes('[COMPLETE')); + expect(premature).toEqual([]); + + conn.finishResponse(holdId); + // The injector flushes the whole queue as ONE combined context item: + // both job conclusions must land in the same conversation.item.create. + const merged = await stack.fakeDash.waitForMessage( + (message) => { + const text = contextTextOf(message); + return text !== undefined && text.includes(`[COMPLETE ${jobA}]`); + }, + { + timeoutMs: 15_000, + fromIndex: inboxIndex, + description: `the merged [COMPLETE ${jobA}]/[COMPLETE ${jobB}] injection`, + }, + ); + const mergedText = contextTextOf(merged)!; + expect(mergedText).toContain(`[COMPLETE ${jobA}]`); + expect(mergedText).toContain(`[COMPLETE ${jobB}]`); + }); +}); diff --git a/integration-tests/qwen-live-m2-permission.test.ts b/integration-tests/qwen-live-m2-permission.test.ts new file mode 100644 index 00000000000..045d15689ba --- /dev/null +++ b/integration-tests/qwen-live-m2-permission.test.ts @@ -0,0 +1,211 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * qwen-live M2 — permission forwarding across the voice boundary: + * + * A real `qwen serve` session (approval mode `default`) hits a `write_file` + * tool call, which raises a daemon permission_request. The qwen-live + * orchestrator must inject a `[PERMISSION req_1]` context item plus a spoken + * ask into the realtime conversation; the voice model's + * `respond_permission {decision:"allow"}` call must be delivered back to + * serve as a vote, after which the tool runs and the turn completes. + * + * A warmup handoff first materializes the orchestrator's default session so + * the test can pin its approval mode deterministically before the + * permission-triggering turn. + */ + +import { readFileSync } from 'node:fs'; +import * as path from 'node:path'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { fakeToolCall } from './fake-openai-server.js'; +import { + contextTextOf, + functionCallOutputOf, + type FakeDashScopeConnection, +} from './fake-dashscope-server.js'; +import { + bootLiveStack, + startLiveCall, + type LiveStack, +} from './qwen-live-harness.js'; + +const SKIP = + process.platform === 'win32' || + Boolean( + process.env['QWEN_SANDBOX'] && + process.env['QWEN_SANDBOX'].toLowerCase() !== 'false', + ); +const describeE2E = SKIP ? describe.skip : describe; + +const PERM_FILE_NAME = 'perm-e2e.txt'; +const PERM_FILE_CONTENT = 'permission-granted'; + +describeE2E('qwen-live M2 — permission relay', () => { + let stack: LiveStack; + let conn: FakeDashScopeConnection; + let permFilePath = ''; + + beforeAll(async () => { + stack = await bootLiveStack({ + makeOpenAIHandler: ({ workspaceDir }) => { + permFilePath = path.join(workspaceDir, PERM_FILE_NAME); + return ({ body }) => { + const messages = JSON.stringify(body['messages'] ?? []); + const hasToolResult = + messages.includes('"role":"tool"') || + messages.includes('"tool_call_id"'); + if (messages.includes('perm-write-task')) { + if (!hasToolResult) { + return { + toolCalls: [ + fakeToolCall('write_file', { + file_path: permFilePath, + content: PERM_FILE_CONTENT, + }), + ], + }; + } + return { content: 'permission turn complete' }; + } + if (messages.includes('perm-warmup')) { + return { content: 'warmup done' }; + } + return { content: 'ok' }; + }; + }, + }); + conn = (await startLiveCall(stack)).conn; + }, 180_000); + + afterAll(async () => { + await stack?.dispose(); + }, 60_000); + + it('relays the permission ask to voice and delivers the allow vote', async () => { + // Warmup: materialize the orchestrator's default serve session. + conn.functionCall({ + name: 'handoff', + argumentsJson: JSON.stringify({ task: 'perm-warmup' }), + callId: 'call-w', + }); + const warmupReceiptMessage = await stack.fakeDash.waitForMessage( + (message) => functionCallOutputOf(message)?.callId === 'call-w', + { timeoutMs: 30_000, description: 'the warmup handoff receipt' }, + ); + const warmupReceipt = JSON.parse( + functionCallOutputOf(warmupReceiptMessage)!.output, + ) as Record; + expect(warmupReceipt['status']).toBe('accepted'); + const warmupJob = String(warmupReceipt['job']); + await stack.fakeDash.waitForMessage( + (message) => + contextTextOf(message)?.includes(`[COMPLETE ${warmupJob}]`) ?? false, + { timeoutMs: 30_000, description: 'the warmup [COMPLETE] injection' }, + ); + + // Pin the approval mode of the orchestrator-created session so the + // write below deterministically raises a permission_request. + const sessions = await stack.serve.client.listWorkspaceSessions( + stack.workspaceDir, + ); + const target = sessions.find( + (session) => session.sessionId !== stack.prewarmSessionId, + ); + expect(target).toBeDefined(); + await stack.serve.client.setSessionApprovalMode( + target!.sessionId, + 'default', + ); + + // The permission-triggering handoff. + const inboxIndex = stack.fakeDash.inbox.length; + conn.functionCall({ + name: 'handoff', + argumentsJson: JSON.stringify({ task: 'perm-write-task' }), + callId: 'call-p', + }); + const receiptMessage = await stack.fakeDash.waitForMessage( + (message) => functionCallOutputOf(message)?.callId === 'call-p', + { + timeoutMs: 30_000, + fromIndex: inboxIndex, + description: 'the perm-write-task handoff receipt', + }, + ); + const receipt = JSON.parse( + functionCallOutputOf(receiptMessage)!.output, + ) as Record; + expect(receipt['status']).toBe('accepted'); + const job = String(receipt['job']); + + // The ask reaches the voice model: silent [PERMISSION req_1] context… + const permissionMessage = await stack.fakeDash.waitForMessage( + (message) => + contextTextOf(message)?.includes('[PERMISSION req_1]') ?? false, + { + timeoutMs: 30_000, + fromIndex: inboxIndex, + description: 'the [PERMISSION req_1] context injection', + }, + ); + const permissionText = contextTextOf(permissionMessage)!; + expect(permissionText).toContain('respond_permission'); + expect(permissionText).toContain('wants to run'); + // …plus a spoken ask ([SPEAK_TO_USER] speech request). + const speakMessage = await stack.fakeDash.waitForMessage( + (message) => { + const text = contextTextOf(message); + return ( + text !== undefined && + text.startsWith('[SPEAK_TO_USER] ') && + text.includes('Should I allow it?') + ); + }, + { + timeoutMs: 15_000, + fromIndex: inboxIndex, + description: 'the spoken permission ask', + }, + ); + expect(contextTextOf(speakMessage)).toBeDefined(); + + // The user says yes: respond_permission must deliver the vote to serve. + conn.functionCall({ + name: 'respond_permission', + argumentsJson: '{"request_id":"req_1","decision":"allow"}', + callId: 'call-2', + }); + const voteReceiptMessage = await stack.fakeDash.waitForMessage( + (message) => functionCallOutputOf(message)?.callId === 'call-2', + { + timeoutMs: 15_000, + fromIndex: inboxIndex, + description: 'the respond_permission receipt', + }, + ); + const voteReceipt = JSON.parse( + functionCallOutputOf(voteReceiptMessage)!.output, + ) as Record; + expect(voteReceipt['status']).toBe('delivered'); + + // Serve accepted the vote: the tool ran and the turn completed. + const completeMessage = await stack.fakeDash.waitForMessage( + (message) => + contextTextOf(message)?.includes(`[COMPLETE ${job}]`) ?? false, + { + timeoutMs: 60_000, + fromIndex: inboxIndex, + description: `[COMPLETE ${job}] after the allow vote`, + }, + ); + expect(contextTextOf(completeMessage)).toContain( + 'permission turn complete', + ); + expect(readFileSync(permFilePath, 'utf8')).toBe(PERM_FILE_CONTENT); + }); +}); diff --git a/integration-tests/qwen-live-m2-steering.test.ts b/integration-tests/qwen-live-m2-steering.test.ts new file mode 100644 index 00000000000..64ce227b351 --- /dev/null +++ b/integration-tests/qwen-live-m2-steering.test.ts @@ -0,0 +1,208 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * qwen-live M2 — steering semantics of `handoff` receipts: + * + * 1. while a serve turn is running (the fake OpenAI handler holds the + * model response on a gate), a second handoff targeting the same + * session is delivered as a mid-turn message — the receipt reads + * `status:"accepted"` with the 'joined the currently running task' + * note (qwen-code-adaptor's steer path); + * 2. once the session is idle again, a handoff with an explicit `session` + * argument is a plain accepted prompt (new job, no 'joined' note). + */ + +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { sleep } from './cli/_daemon-harness.js'; +import { + contextTextOf, + functionCallOutputOf, + type FakeDashScopeConnection, +} from './fake-dashscope-server.js'; +import { + bootLiveStack, + deferred, + startLiveCall, + withTimeout, + type Deferred, + type LiveStack, +} from './qwen-live-harness.js'; + +const SKIP = + process.platform === 'win32' || + Boolean( + process.env['QWEN_SANDBOX'] && + process.env['QWEN_SANDBOX'].toLowerCase() !== 'false', + ); +const describeE2E = SKIP ? describe.skip : describe; + +const SLOW_MARKER = 'steer-slow-task'; + +describeE2E('qwen-live M2 — mid-turn steering', () => { + let stack: LiveStack; + let conn: FakeDashScopeConnection; + let sessionHandle = ''; + const slowGate: Deferred = deferred(); + const slowRequestSeen: Deferred = deferred(); + /** Serialized `messages` of every model request the fake endpoint saw. */ + const modelRequests: string[] = []; + + const toolCall = async ( + name: string, + callId: string, + args: Record, + ): Promise> => { + conn.functionCall({ + name, + argumentsJson: JSON.stringify(args), + callId, + }); + const receiptMessage = await stack.fakeDash.waitForMessage( + (message) => functionCallOutputOf(message)?.callId === callId, + { timeoutMs: 30_000, description: `${name} receipt ${callId}` }, + ); + return JSON.parse(functionCallOutputOf(receiptMessage)!.output) as Record< + string, + unknown + >; + }; + + const handoff = ( + callId: string, + args: Record, + ): Promise> => toolCall('handoff', callId, args); + + /** + * Poll the session through the protocol's own probe (session_monitor) + * until the orchestrator reports it idle. A steered mid-turn message can + * legitimately be promoted into a follow-up turn after the first one + * completes, so "the [COMPLETE] arrived" does not imply "idle". + */ + let monitorSeq = 0; + const waitForIdleSession = async ( + session: string, + timeoutMs = 30_000, + ): Promise => { + const deadline = Date.now() + timeoutMs; + for (;;) { + const snapshot = await toolCall( + 'session_monitor', + `call-mon-${++monitorSeq}`, + { + session, + }, + ); + expect(snapshot['status']).toBe('ok'); + if (snapshot['state'] === 'idle') return; + if (Date.now() > deadline) { + throw new Error( + `session ${session} still ${String(snapshot['state'])} after ${timeoutMs}ms`, + ); + } + await sleep(200); + } + }; + + beforeAll(async () => { + stack = await bootLiveStack({ + makeOpenAIHandler: + () => + async ({ body }) => { + const messages = JSON.stringify(body['messages'] ?? []); + modelRequests.push(messages); + if (messages.includes(SLOW_MARKER)) { + slowRequestSeen.resolve(); + await slowGate.promise; // hang the turn until the test releases it + return { content: 'slow task finished' }; + } + return { content: 'ok' }; + }, + }); + conn = (await startLiveCall(stack)).conn; + }, 180_000); + + afterAll(async () => { + slowGate.resolve(); // never leave serve hung on teardown + await stack?.dispose(); + }, 60_000); + + it('joins the running turn when handing off to a busy session', async () => { + const first = await handoff('call-s1', { task: SLOW_MARKER }); + expect(first['status']).toBe('accepted'); + expect(typeof first['job']).toBe('string'); + sessionHandle = String(first['session']); + expect(sessionHandle).toMatch(/^session_\d+$/); + + // The serve turn is really running: the model request is parked on the + // gate inside the fake OpenAI handler. + await withTimeout( + slowRequestSeen.promise, + 30_000, + 'the slow turn to reach the fake model endpoint', + ); + + const second = await handoff('call-s2', { + task: 'also update the changelog', + session: sessionHandle, + }); + expect(second['status']).toBe('accepted'); + expect(second['session']).toBe(sessionHandle); + expect(String(second['note'])).toContain('joined'); + + // Release the turn; its conclusion must flow back as [COMPLETE]. + const inboxIndex = stack.fakeDash.inbox.length; + slowGate.resolve(); + const complete = await stack.fakeDash.waitForMessage( + (message) => { + const text = contextTextOf(message); + return text !== undefined && /\[COMPLETE job_\d+\]/.test(text); + }, + { + timeoutMs: 30_000, + fromIndex: inboxIndex, + description: 'the [COMPLETE] injection for the steered turn', + }, + ); + expect(contextTextOf(complete)).toContain('slow task finished'); + }); + + it('accepts a plain handoff to the now-idle session', async () => { + // The joined instruction may be promoted into a follow-up turn after the + // first turn completes; wait until the orchestrator reports idle. + await waitForIdleSession(sessionHandle); + + // The 'joined' receipt was not just wording: by the time the session is + // idle again, the steered instruction must have reached the model in + // some request (injected mid-turn or promoted to a follow-up turn). + expect( + modelRequests.some((request) => + request.includes('also update the changelog'), + ), + ).toBe(true); + + const receipt = await handoff('call-s3', { + task: 'one more quick task', + session: sessionHandle, + }); + expect(receipt['status']).toBe('accepted'); + expect(receipt['session']).toBe(sessionHandle); + expect(typeof receipt['job']).toBe('string'); + // Idle session ⇒ a fresh prompt, not a mid-turn join. + expect(String(receipt['note'] ?? '')).not.toContain('joined'); + expect(String(receipt['note'] ?? '')).not.toContain('queued'); + + const job = String(receipt['job']); + await stack.fakeDash.waitForMessage( + (message) => + contextTextOf(message)?.includes(`[COMPLETE ${job}]`) ?? false, + { + timeoutMs: 30_000, + description: `[COMPLETE ${job}] for the idle handoff`, + }, + ); + }); +}); diff --git a/package-lock.json b/package-lock.json index 7aca721fc92..d9dbd318ec3 100644 --- a/package-lock.json +++ b/package-lock.json @@ -4048,6 +4048,9 @@ "x64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -4700,6 +4703,9 @@ "arm64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -4717,6 +4723,9 @@ "arm64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -4734,6 +4743,9 @@ "ppc64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -4751,6 +4763,9 @@ "riscv64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -4768,6 +4783,9 @@ "riscv64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -4785,6 +4803,9 @@ "s390x" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -4802,6 +4823,9 @@ "x64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -4819,6 +4843,9 @@ "x64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -5042,6 +5069,9 @@ "arm64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -5056,6 +5086,9 @@ "arm64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -5070,6 +5103,9 @@ "ppc64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -5084,6 +5120,9 @@ "riscv64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -5098,6 +5137,9 @@ "riscv64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -5112,6 +5154,9 @@ "s390x" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -5126,6 +5171,9 @@ "x64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -5140,6 +5188,9 @@ "x64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -7121,6 +7172,9 @@ "arm" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -7135,6 +7189,9 @@ "arm" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -7149,6 +7206,9 @@ "arm64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -7163,6 +7223,9 @@ "arm64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -7177,6 +7240,9 @@ "loong64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -7191,6 +7257,9 @@ "loong64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -7205,6 +7274,9 @@ "ppc64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -7219,6 +7291,9 @@ "ppc64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -7233,6 +7308,9 @@ "riscv64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -7247,6 +7325,9 @@ "riscv64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -7261,6 +7342,9 @@ "s390x" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -7275,6 +7359,9 @@ "x64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "MIT", "optional": true, "os": [ @@ -7289,6 +7376,9 @@ "x64" ], "dev": true, + "libc": [ + "musl" + ], "license": "MIT", "optional": true, "os": [ @@ -35231,6 +35321,44 @@ "optional": true } } + }, + "node_modules/@qwen-code/qwen-live": { + "resolved": "packages/qwen-live", + "link": true + }, + "packages/qwen-live": { + "name": "@qwen-code/qwen-live", + "version": "0.1.0", + "license": "Apache-2.0", + "dependencies": { + "@qwen-code/sdk": "file:../sdk-typescript", + "ansi-regex": "^6.2.2", + "proper-lockfile": "^4.1.2", + "ws": "^8.18.0" + }, + "bin": { + "qwen-live": "dist/index.js" + }, + "devDependencies": { + "@types/node": "^22.13.10", + "@types/proper-lockfile": "^4.1.4", + "@types/ws": "^8.5.0", + "typescript": "^5.3.3", + "vitest": "^3.1.1" + }, + "engines": { + "node": ">=22" + } + }, + "packages/qwen-live/node_modules/@types/node": { + "version": "22.19.19", + "resolved": "https://registry.npmjs.org/@types/node/-/node-22.19.19.tgz", + "integrity": "sha512-dyh/xO2Fh5bYrfWaaqGrRQQGkNdmYw6AmaAUvYeUMNTWQtvb796ikLdmTchRmOlOiIJ1TDXfWgVx1QkUlQ6Hew==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": "~6.21.0" + } } } } diff --git a/package.json b/package.json index 2f43c0aec1a..520110918a0 100644 --- a/package.json +++ b/package.json @@ -55,7 +55,7 @@ "test:integration:sandbox:none": "cross-env QWEN_SANDBOX=false vitest run --root ./integration-tests", "test:integration:sandbox:docker": "cross-env QWEN_SANDBOX=docker npm run build:sandbox && QWEN_SANDBOX=docker vitest run --root ./integration-tests", "test:integration:sandbox:podman": "cross-env QWEN_SANDBOX=podman vitest run --root ./integration-tests", - "test:integration:no-ak:sandbox:none": "cross-env QWEN_SANDBOX=false vitest run --root ./integration-tests --poolOptions.forks.maxForks 2 ./fake-openai-server.test.ts ./test-helper.test.ts ./chat-transcript-contract.test.ts ./cli/daemon-invocation-context.test.ts ./cli/list_directory.test.ts ./cli/qwen-serve-routes.test.ts ./cli/qwen-serve-streaming.test.ts ./sdk-typescript/abort-and-lifecycle.test.ts ./sdk-typescript/permission-control.test.ts ./sdk-typescript/sdk-mcp-server.test.ts ./sdk-typescript/subagents.test.ts ./sdk-typescript/system-control.test.ts ./sdk-typescript/tool-control.test.ts", + "test:integration:no-ak:sandbox:none": "cross-env QWEN_SANDBOX=false vitest run --root ./integration-tests --poolOptions.forks.maxForks 2 ./fake-openai-server.test.ts ./test-helper.test.ts ./chat-transcript-contract.test.ts ./qwen-live-m1-call.test.ts ./qwen-live-m2-inject.test.ts ./qwen-live-m2-permission.test.ts ./qwen-live-m2-steering.test.ts ./cli/daemon-invocation-context.test.ts ./cli/list_directory.test.ts ./cli/qwen-serve-routes.test.ts ./cli/qwen-serve-streaming.test.ts ./sdk-typescript/abort-and-lifecycle.test.ts ./sdk-typescript/permission-control.test.ts ./sdk-typescript/sdk-mcp-server.test.ts ./sdk-typescript/subagents.test.ts ./sdk-typescript/system-control.test.ts ./sdk-typescript/tool-control.test.ts", "test:integration:sdk:sandbox:none": "cross-env QWEN_SANDBOX=false vitest run --root ./integration-tests --poolOptions.forks.maxForks 2 sdk-typescript", "test:integration:sdk:sandbox:docker": "cross-env QWEN_SANDBOX=docker npm run build:sandbox && QWEN_SANDBOX=docker vitest run --root ./integration-tests --poolOptions.forks.maxForks 2 sdk-typescript", "test:sdk:python": "python3 -m pytest -c packages/sdk-python/pyproject.toml packages/sdk-python/tests -q", diff --git a/packages/cli/src/serve/live/live-host-installer.ts b/packages/cli/src/serve/live/live-host-installer.ts index 870b65897c6..97c72b70b84 100644 --- a/packages/cli/src/serve/live/live-host-installer.ts +++ b/packages/cli/src/serve/live/live-host-installer.ts @@ -457,6 +457,9 @@ export class LiveHostInstaller { } async launch(): Promise { + if (this.platform !== 'darwin') { + return this.setError('Qwen Live Host is available only on macOS.', false); + } if (this.operation) return await this.operation; try { const installed = await this.inspectInstalled(); diff --git a/packages/core/src/tools/tool-registry.test.ts b/packages/core/src/tools/tool-registry.test.ts index aa137b022f4..42b2663eb19 100644 --- a/packages/core/src/tools/tool-registry.test.ts +++ b/packages/core/src/tools/tool-registry.test.ts @@ -947,9 +947,9 @@ describe('ToolRegistry', () => { 'hidden_by_allowlist', ); expect(registry.isDeferredAndHidden('hidden_by_allowlist')).toBe(false); - expect(registry.getDeferredToolSummary().map((t) => t.name)).not.toContain( - 'hidden_by_allowlist', - ); + expect( + registry.getDeferredToolSummary().map((t) => t.name), + ).not.toContain('hidden_by_allowlist'); }); it('reveals the schema once ToolSearch loads the tool', async () => { diff --git a/packages/live-host/src/main/__tests__/protocol.test.ts b/packages/live-host/src/main/__tests__/protocol.test.ts index f0cc57f6fd0..4be7e44fd92 100644 --- a/packages/live-host/src/main/__tests__/protocol.test.ts +++ b/packages/live-host/src/main/__tests__/protocol.test.ts @@ -113,7 +113,23 @@ const DAEMON_PROTOCOL_TYPES_URL = new URL( import.meta.url, ); +const QWEN_LIVE_PROTOCOL_TYPES_URL = new URL( + '../../../../qwen-live/src/host/types.ts', + import.meta.url, +); + describe('Live Host protocol', () => { + it('keeps the qwen-live daemon contract byte-identical to the cli copy', async () => { + // PROTOCOL_TYPE_PARITY type-checks against the cli copy only; the + // standalone qwen-live daemon validates and emits against its own copy. + // Byte-identity keeps a change to either copy from drifting silently. + const [cliSource, qwenLiveSource] = await Promise.all([ + readFile(fileURLToPath(DAEMON_PROTOCOL_TYPES_URL), 'utf8'), + readFile(fileURLToPath(QWEN_LIVE_PROTOCOL_TYPES_URL), 'utf8'), + ]); + assert.equal(qwenLiveSource, cliSource); + }); + it('stays synchronized with the daemon protocol contract', async () => { const source = await readFile( fileURLToPath(DAEMON_PROTOCOL_TYPES_URL), diff --git a/packages/qwen-live/.gitignore b/packages/qwen-live/.gitignore new file mode 100644 index 00000000000..4bf7b955a97 --- /dev/null +++ b/packages/qwen-live/.gitignore @@ -0,0 +1,2 @@ +dist/ +*.tsbuildinfo diff --git a/packages/qwen-live/README.md b/packages/qwen-live/README.md new file mode 100644 index 00000000000..cd1a8f29902 --- /dev/null +++ b/packages/qwen-live/README.md @@ -0,0 +1,57 @@ +# @qwen-code/qwen-live + +Standalone Live voice daemon: a realtime voice control plane that orchestrates +Qwen Code sessions. + +`qwen-live` connects three parties: + +- **Qwen Live Host** (the macOS overlay app) over the Live Host WebSocket + protocol v6 — the same wire protocol and `~/.qwen/live/daemon.json` discovery + file the built-in `qwen serve` Live integration uses, so an already-installed + Host binary connects without changes. +- **A DashScope realtime voice model** (`qwen-omni` realtime) that owns the + conversation: VAD, direct answers, and a small tool surface for dispatching + work. +- **Qwen Code sessions** through a `BackendAdaptor`. The first adaptor drives a + running `qwen serve` daemon over its REST/SSE surface. + +The live session itself is fully owned by this daemon (JSONL logs under +`~/.qwen-live/sessions/`); backend sessions are ordinary Qwen Code sessions +that keep running after a call ends. + +## Run + +```bash +# Requires a running `qwen serve` and a DashScope API key. +DASHSCOPE_API_KEY=sk-... qwen-live +``` + +Configuration comes from environment variables (`QWEN_LIVE_*`, +`DASHSCOPE_API_KEY`, `QWEN_SERVER_TOKEN`) with `~/.qwen-live/config.json` as +the file-based fallback. See `src/config.ts` for the full list. + +Only one Live daemon may own the Host discovery file at a time. If the +built-in `qwen serve` Live integration is enabled and running, `qwen-live` +fails fast at startup instead of taking over. + +## Host bootstrap + +The Live Host installer is ported: `GET /live/setup` reports the installed +Host's state, `POST /live/setup/install` downloads, verifies (sha256 + +codesign + team identifier), and installs the latest Host release, and +`POST /live/setup/launch` opens it. macOS only (the same surface `qwen +serve` exposes). All three sit behind the daemon's Bearer token; a browser +context (Origin header) is always refused. + +```bash +TOKEN=$(jq -r .token ~/.qwen/live/daemon.json) +PORT=$(jq -r .url ~/.qwen/live/daemon.json | sed 's/.*://') +curl -H "authorization: Bearer $TOKEN" "http://127.0.0.1:$PORT/live/setup" +``` + +## Status + +Incubating inside the qwen-code monorepo, tracking M1+M2 (plus the M1 +installer, now ported) of the Live split roadmap (issue #10118). The Host app itself, non-qwen-code adaptors (M4), and retiring +the built-in Live integration (M5) are tracked in later milestones of +issue #10118. diff --git a/packages/qwen-live/build.mjs b/packages/qwen-live/build.mjs new file mode 100644 index 00000000000..2e60dbc0aef --- /dev/null +++ b/packages/qwen-live/build.mjs @@ -0,0 +1,41 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Build script for @qwen-code/qwen-live. + * + * Cleans dist plus the incremental build info together (removing dist alone + * leaves tsc's buildinfo claiming everything is up to date, so `tsc --build` + * would emit nothing), compiles TypeScript, and sanity-checks the emit. + */ +import { existsSync, rmSync } from 'node:fs'; +import path from 'node:path'; +import { createRequire } from 'node:module'; +import { fileURLToPath } from 'node:url'; +import { execFileSync } from 'node:child_process'; + +const require = createRequire(import.meta.url); +const here = path.dirname(fileURLToPath(import.meta.url)); + +rmSync(path.join(here, 'dist'), { recursive: true, force: true }); +rmSync(path.join(here, 'tsconfig.build.tsbuildinfo'), { force: true }); + +// Resolve tsc through node rather than `npx` so this also works on Windows, +// where execFileSync cannot resolve `npx.cmd` without a shell. +execFileSync( + process.execPath, + [require.resolve('typescript/bin/tsc'), '--build', 'tsconfig.build.json'], + { cwd: here, stdio: 'inherit' }, +); + +for (const required of ['index.js', 'daemon.js']) { + const emitted = path.join(here, 'dist', required); + if (!existsSync(emitted)) { + throw new Error( + `build produced no dist/${required} — tsc emitted nothing (stale tsconfig.build.tsbuildinfo?)`, + ); + } +} diff --git a/packages/qwen-live/package.json b/packages/qwen-live/package.json new file mode 100644 index 00000000000..64ef243901f --- /dev/null +++ b/packages/qwen-live/package.json @@ -0,0 +1,57 @@ +{ + "name": "@qwen-code/qwen-live", + "version": "0.1.0", + "description": "Standalone Live voice daemon: a realtime voice control plane that orchestrates Qwen Code sessions.", + "repository": { + "type": "git", + "url": "git+https://github.com/QwenLM/qwen-code.git", + "directory": "packages/qwen-live" + }, + "type": "module", + "license": "Apache-2.0", + "engines": { + "node": ">=22" + }, + "main": "dist/index.js", + "types": "dist/index.d.ts", + "bin": { + "qwen-live": "dist/index.js" + }, + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js" + }, + "./package.json": "./package.json" + }, + "files": [ + "dist", + "!dist/**/*.test.*" + ], + "scripts": { + "build": "node build.mjs", + "build:ts": "tsc --build tsconfig.build.json", + "clean": "rm -rf dist tsconfig.tsbuildinfo tsconfig.build.tsbuildinfo", + "test": "vitest run", + "test:ci": "vitest run", + "typecheck": "tsc --noEmit", + "watch": "tsc --watch" + }, + "dependencies": { + "@qwen-code/sdk": "file:../sdk-typescript", + "ansi-regex": "^6.2.2", + "proper-lockfile": "^4.1.2", + "ws": "^8.18.0" + }, + "devDependencies": { + "@types/node": "^22.13.10", + "@types/proper-lockfile": "^4.1.4", + "@types/ws": "^8.5.0", + "typescript": "^5.3.3", + "vitest": "^3.1.1" + }, + "bugs": { + "url": "https://github.com/QwenLM/qwen-code/issues" + }, + "homepage": "https://github.com/QwenLM/qwen-code/tree/main/packages/qwen-live#readme" +} diff --git a/packages/qwen-live/src/adaptor/qwen-code-adaptor.test.ts b/packages/qwen-live/src/adaptor/qwen-code-adaptor.test.ts new file mode 100644 index 00000000000..2cff900d154 --- /dev/null +++ b/packages/qwen-live/src/adaptor/qwen-code-adaptor.test.ts @@ -0,0 +1,1201 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it, vi } from 'vitest'; +import { QwenCodeAdaptor } from './qwen-code-adaptor.js'; +import type { + DaemonClientLike, + QwenCodeAdaptorOptions, +} from './qwen-code-adaptor.js'; +import type { BackendEvent, BackendHandle, ContentBlock } from './types.js'; + +/** Mirrors REQUIRED_FEATURES in qwen-code-adaptor.ts. */ +const ALL_FEATURES = [ + 'session_create', + 'session_prompt', + 'session_events', + 'session_cancel', + 'session_permission_vote', + 'session_mid_turn_message_mutation', +] as const; + +const CLIENT_ID = 'live-client-1'; +// The id the daemon issues on create/attach; per-session calls must echo it. +const ISSUED_CLIENT_ID = 'daemon-issued-7'; +const BASE_URL = 'http://127.0.0.1:0'; +const SESSION_ID = 'sess-1'; + +interface EventEnvelope { + id?: number; + v: 1; + type: string; + data: unknown; + promptId?: string; + originatorClientId?: string; +} + +type EnvelopeStream = ReturnType; + +function envelope( + type: string, + data: unknown = {}, + extra: Partial> = {}, +): EventEnvelope { + return { v: 1, type, data, ...extra }; +} + +async function* envelopeStream( + envelopes: readonly EventEnvelope[], +): EnvelopeStream { + for (const item of envelopes) yield item; +} + +function makeClient( + overrides: Partial = {}, +): DaemonClientLike { + return { + capabilities: vi.fn(async () => ({ + features: [...ALL_FEATURES], + workspaceCwd: '/daemon-ws', + })), + createOrAttachSession: vi.fn(async () => ({ + sessionId: SESSION_ID, + clientId: ISSUED_CLIENT_ID, + })), + listWorkspaceSessions: vi.fn(async () => []), + promptNonBlocking: vi.fn(async () => ({ promptId: 'p1' })), + subscribeEvents: vi.fn(() => envelopeStream([])), + enqueueMidTurnMessage: vi.fn(async () => ({ accepted: true })), + cancel: vi.fn(async () => undefined), + respondToSessionPermission: vi.fn(async () => true), + uploadSessionAttachment: vi.fn(async () => ({ + type: 'resource_link', + uri: 'attachment://shot', + })), + updateSessionMetadata: vi.fn(async () => ({})), + ...overrides, + }; +} + +function makeAdaptor( + client: DaemonClientLike, + options: Partial = {}, +): QwenCodeAdaptor { + return new QwenCodeAdaptor({ + baseUrl: BASE_URL, + client, + clientId: CLIENT_ID, + defaultCwd: '/ws', + ...options, + }); +} + +function handleFor(id: string = SESSION_ID): BackendHandle { + return { id, adaptor: 'qwen-code' }; +} + +async function collect( + adaptor: QwenCodeAdaptor, + handle: BackendHandle, +): Promise { + const events: BackendEvent[] = []; + for await (const event of adaptor.events(handle)) events.push(event); + return events; +} + +function permissionRequestEnvelope(requestId = 'req-1'): EventEnvelope { + return envelope('permission_request', { + requestId, + toolCall: { name: 'Bash', command: 'rm -rf /tmp' }, + // The ACP wire field for display text is `name` (there is no `label`). + options: [{ optionId: 'allow', name: 'Allow once' }, { optionId: 'deny' }], + }); +} + +/** Feeds one permission_request through the event stream to seed options. */ +async function seedPermissionOptions( + adaptor: QwenCodeAdaptor, + client: DaemonClientLike, + requestId = 'req-1', +): Promise { + await adaptor.createSession(); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([permissionRequestEnvelope(requestId)]), + ); + await collect(adaptor, handleFor()); +} + +describe('QwenCodeAdaptor.preflight', () => { + it('throws listing the missing capabilities', async () => { + const client = makeClient({ + capabilities: vi.fn(async () => ({ + features: ALL_FEATURES.filter( + (feature) => + feature !== 'session_cancel' && + feature !== 'session_mid_turn_message_mutation', + ), + workspaceCwd: '/daemon-ws', + })), + }); + const adaptor = makeAdaptor(client); + + await expect(adaptor.preflight()).rejects.toThrow( + /missing required capabilities.*session_cancel, session_mid_turn_message_mutation/, + ); + }); + + it('wraps network errors with the baseUrl and a start hint', async () => { + const client = makeClient({ + capabilities: vi.fn(async () => { + throw new Error('connect ECONNREFUSED'); + }), + }); + const adaptor = makeAdaptor(client); + + const error = await adaptor.preflight().then( + () => { + throw new Error('preflight resolved unexpectedly'); + }, + (raised: unknown) => raised as Error, + ); + expect(error.message).toContain(BASE_URL); + expect(error.message).toContain('connect ECONNREFUSED'); + expect(error.message).toContain('qwen serve'); + }); + + it('distinguishes an HTTP refusal from an unreachable daemon', async () => { + const client = makeClient({ + capabilities: vi.fn(async () => { + throw Object.assign(new Error('Unauthorized'), { + name: 'DaemonHttpError', + status: 401, + }); + }), + }); + const adaptor = makeAdaptor(client); + + const error = await adaptor.preflight().then( + () => { + throw new Error('preflight resolved unexpectedly'); + }, + (raised: unknown) => raised as Error, + ); + // The daemon responded — telling the operator to start one misdirects. + expect(error.message).not.toContain('not reachable'); + expect(error.message).not.toContain('Start it'); + expect(error.message).toContain('401'); + expect(error.message).toContain('token'); + expect(error.message).toContain(BASE_URL); + }); + + it('remembers the daemon workspaceCwd for later createSession calls', async () => { + const client = makeClient(); + const adaptor = new QwenCodeAdaptor({ + baseUrl: BASE_URL, + client, + clientId: CLIENT_ID, + // No defaultCwd: the preflight-discovered workspaceCwd must win. + }); + + await adaptor.preflight(); + await adaptor.createSession(); + + expect(client.createOrAttachSession).toHaveBeenCalledWith( + expect.objectContaining({ workspaceCwd: '/daemon-ws' }), + CLIENT_ID, + ); + }); +}); + +describe('QwenCodeAdaptor.createSession', () => { + it('requests a thread-scoped qwen-live session and returns the handle', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + + const handle = await adaptor.createSession(); + + expect(client.createOrAttachSession).toHaveBeenCalledWith( + { + workspaceCwd: '/ws', + sessionScope: 'thread', + sourceType: 'qwen-live', + }, + CLIENT_ID, + ); + expect(handle).toEqual({ id: SESSION_ID, adaptor: 'qwen-code' }); + expect(adaptor.isBusy(handle)).toBe(false); + // No label requested: no metadata round-trip. + expect(client.updateSessionMetadata).not.toHaveBeenCalled(); + }); + + it('applies the requested label via a session metadata update', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + + await adaptor.createSession({ label: 'payroll refactor' }); + + expect(client.updateSessionMetadata).toHaveBeenCalledWith( + SESSION_ID, + { displayName: 'payroll refactor' }, + ISSUED_CLIENT_ID, + ); + }); + + it('marks the session busy immediately when hasActivePrompt is set', async () => { + const client = makeClient({ + createOrAttachSession: vi.fn(async () => ({ + sessionId: SESSION_ID, + hasActivePrompt: true, + })), + }); + const adaptor = makeAdaptor(client); + + const handle = await adaptor.createSession(); + + expect(adaptor.isBusy(handle)).toBe(true); + }); +}); + +describe('QwenCodeAdaptor.listSessions', () => { + it('requests a full catalog page instead of the SDK default first page', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + + await adaptor.listSessions(); + + expect(client.listWorkspaceSessions).toHaveBeenCalledWith('/ws', { + pageSize: 1000, + }); + }); + + it('queries every cwd sessions were created in, deduping by sessionId', async () => { + const client = makeClient({ + listWorkspaceSessions: vi.fn(async (cwd: string) => + cwd === '/other' + ? [{ sessionId: 'sess-other' }, { sessionId: SESSION_ID }] + : [{ sessionId: SESSION_ID, displayName: 'main' }], + ), + }); + const adaptor = makeAdaptor(client); + vi.mocked(client.createOrAttachSession).mockResolvedValueOnce({ + sessionId: 'sess-other', + clientId: ISSUED_CLIENT_ID, + }); + await adaptor.createSession({ cwd: '/other' }); + + const sessions = await adaptor.listSessions(); + + const queried = vi + .mocked(client.listWorkspaceSessions) + .mock.calls.map((call) => call[0]); + expect(queried).toContain('/ws'); + expect(queried).toContain('/other'); + expect(sessions.map((s) => s.handle.id).sort()).toEqual([ + SESSION_ID, + 'sess-other', + ]); + // The duplicate row from '/other' must not clobber the first mapping. + expect(sessions.find((s) => s.handle.id === SESSION_ID)?.label).toBe( + 'main', + ); + expect(sessions.find((s) => s.handle.id === 'sess-other')?.cwd).toBe( + '/other', + ); + }); + + it('derives busy/idle for untracked sessions from hasActivePrompt', async () => { + const client = makeClient({ + listWorkspaceSessions: vi.fn(async () => [ + { sessionId: 'a', hasActivePrompt: true }, + { sessionId: 'b', hasActivePrompt: false }, + { sessionId: 'c' }, + ]), + }); + const adaptor = makeAdaptor(client); + + const sessions = await adaptor.listSessions(); + + expect(sessions.map((s) => s.state)).toEqual(['busy', 'idle', 'unknown']); + }); +}); + +describe('QwenCodeAdaptor.prompt', () => { + it('converts text blocks, returns an accepted receipt, and turns busy', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + const handle = await adaptor.createSession(); + + const receipt = await adaptor.prompt(handle, [ + { type: 'text', text: 'run the tests' }, + ]); + + expect(client.promptNonBlocking).toHaveBeenCalledWith( + SESSION_ID, + { prompt: [{ type: 'text', text: 'run the tests' }] }, + undefined, + ISSUED_CLIENT_ID, + ); + expect(receipt).toEqual({ status: 'accepted', jobRef: 'p1' }); + expect(adaptor.isBusy(handle)).toBe(true); + }); + + it('maps a 503 rejection to a rejected receipt with a busy note', async () => { + const client = makeClient({ + promptNonBlocking: vi.fn(async () => { + throw Object.assign(new Error('queue full'), { status: 503 }); + }), + }); + const adaptor = makeAdaptor(client); + const handle = await adaptor.createSession(); + + const receipt = await adaptor.prompt(handle, [ + { type: 'text', text: 'hi' }, + ]); + + expect(receipt.status).toBe('rejected'); + expect(receipt.note).toContain('busy'); + expect(adaptor.isBusy(handle)).toBe(false); + }); +}); + +describe('QwenCodeAdaptor.prompt steering', () => { + async function busyAdaptor(client: DaemonClientLike) { + vi.mocked(client.createOrAttachSession).mockResolvedValueOnce({ + sessionId: SESSION_ID, + hasActivePrompt: true, + clientId: ISSUED_CLIENT_ID, + }); + const adaptor = makeAdaptor(client); + const handle = await adaptor.createSession(); + return { adaptor, handle }; + } + + it('joins the active turn when the mid-turn injection is accepted', async () => { + const client = makeClient(); + const { adaptor, handle } = await busyAdaptor(client); + + const receipt = await adaptor.prompt( + handle, + [ + { type: 'text', text: 'also update the docs' }, + { type: 'text', text: 'and keep it short' }, + ], + { steer: true }, + ); + + expect(client.enqueueMidTurnMessage).toHaveBeenCalledWith( + SESSION_ID, + 'also update the docs\n\nand keep it short', + { clientId: ISSUED_CLIENT_ID }, + ); + expect(client.promptNonBlocking).not.toHaveBeenCalled(); + expect(receipt.status).toBe('accepted'); + expect(receipt.joinedActiveTurn).toBe(true); + }); + + it('falls back to a queued prompt when the injection is rejected', async () => { + const client = makeClient({ + enqueueMidTurnMessage: vi.fn(async () => ({ accepted: false })), + }); + const { adaptor, handle } = await busyAdaptor(client); + + const receipt = await adaptor.prompt( + handle, + [{ type: 'text', text: 'next up' }], + { steer: true }, + ); + + expect(client.enqueueMidTurnMessage).toHaveBeenCalledOnce(); + expect(client.promptNonBlocking).toHaveBeenCalledOnce(); + expect(receipt.status).toBe('queued'); + expect(receipt.jobRef).toBe('p1'); + expect(receipt.joinedActiveTurn).toBeUndefined(); + }); + + it('falls back to a queued prompt when the injection is rejected with 400', async () => { + // The mid-turn route caps the message length with a 400; the identical + // payload is admissible as a queued full prompt. + const client = makeClient({ + enqueueMidTurnMessage: vi.fn(async () => { + throw Object.assign(new Error('message too long'), { + name: 'DaemonHttpError', + status: 400, + }); + }), + }); + const { adaptor, handle } = await busyAdaptor(client); + + const receipt = await adaptor.prompt( + handle, + [{ type: 'text', text: 'a very long steer' }], + { steer: true }, + ); + + expect(client.enqueueMidTurnMessage).toHaveBeenCalledOnce(); + expect(client.promptNonBlocking).toHaveBeenCalledOnce(); + expect(receipt.status).toBe('queued'); + expect(receipt.joinedActiveTurn).toBeUndefined(); + }); + + it('rethrows non-400 mid-turn injection failures', async () => { + const client = makeClient({ + enqueueMidTurnMessage: vi.fn(async () => { + throw Object.assign(new Error('daemon exploded'), { status: 500 }); + }), + }); + const { adaptor, handle } = await busyAdaptor(client); + + await expect( + adaptor.prompt(handle, [{ type: 'text', text: 'x' }], { steer: true }), + ).rejects.toThrow('daemon exploded'); + expect(client.promptNonBlocking).not.toHaveBeenCalled(); + }); + + it('falls back to a full prompt when a steer payload carries images', async () => { + const client = makeClient(); + const { adaptor, handle } = await busyAdaptor(client); + + const receipt = await adaptor.prompt( + handle, + [ + { type: 'text', text: 'look at this' }, + { + type: 'image', + mimeType: 'image/png', + data: new Uint8Array([1, 2, 3]), + name: 'shot.png', + }, + ], + { steer: true }, + ); + + // Mid-turn injection is text-only; dropping the screenshot silently + // would betray the handoff contract. + expect(client.enqueueMidTurnMessage).not.toHaveBeenCalled(); + expect(client.uploadSessionAttachment).toHaveBeenCalledOnce(); + expect(client.promptNonBlocking).toHaveBeenCalledOnce(); + expect(receipt.status).toBe('queued'); + }); + + it('maps the client-side pending-prompt limit error to a rejected receipt', async () => { + const client = makeClient({ + promptNonBlocking: vi.fn(async () => { + const error = new Error('Pending prompts full: "sess" (2/2)'); + error.name = 'DaemonPendingPromptLimitError'; + throw error; + }), + }); + const { adaptor, handle } = await busyAdaptor(client); + vi.mocked(client.enqueueMidTurnMessage).mockResolvedValueOnce({ + accepted: false, + }); + + const receipt = await adaptor.prompt( + handle, + [{ type: 'text', text: 'one more thing' }], + { steer: true }, + ); + + expect(receipt.status).toBe('rejected'); + expect(receipt.note).toContain('queue is full'); + }); + + it('skips the mid-turn injection entirely when the session is idle', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + const handle = await adaptor.createSession(); + + const receipt = await adaptor.prompt( + handle, + [{ type: 'text', text: 'do the thing' }], + { steer: true }, + ); + + expect(client.enqueueMidTurnMessage).not.toHaveBeenCalled(); + expect(client.promptNonBlocking).toHaveBeenCalledOnce(); + expect(receipt.status).toBe('queued'); + }); +}); + +describe('QwenCodeAdaptor.events', () => { + it('normalizes a full turn: started, buffered chunks, progress, complete', async () => { + const client = makeClient({ + subscribeEvents: vi.fn(() => + envelopeStream([ + envelope('pending_prompt_started', {}, { promptId: 'p1' }), + envelope('session_update', { + update: { + sessionUpdate: 'agent_message_chunk', + content: { type: 'text', text: 'Hello ' }, + }, + }), + envelope('session_update', { + update: { + sessionUpdate: 'agent_message_chunk', + content: { type: 'text', text: 'world' }, + }, + }), + envelope('session_update', { + update: { sessionUpdate: 'tool_call', title: 'Run npm test' }, + }), + envelope('turn_complete', {}, { promptId: 'p1' }), + ]), + ), + }); + const adaptor = makeAdaptor(client); + const handle = handleFor(); + + const iterator = adaptor.events(handle)[Symbol.asyncIterator](); + + const started = await iterator.next(); + expect(started.value).toEqual({ type: 'turn_started', jobRef: 'p1' }); + expect(adaptor.isBusy(handle)).toBe(true); + + // The two agent_message_chunk envelopes yield no events of their own; + // the next observable event is the tool_call progress line. + const progress = await iterator.next(); + expect(progress.value).toEqual({ + type: 'progress', + summary: 'Run npm test', + }); + + const complete = await iterator.next(); + expect(complete.value).toEqual({ + type: 'turn_complete', + jobRef: 'p1', + summary: 'Hello world', + detail: 'Hello world', + }); + expect(adaptor.isBusy(handle)).toBe(false); + + const end = await iterator.next(); + expect(end.done).toBe(true); + }); + + it('maps turn_error to a turn_error event carrying the message', async () => { + const client = makeClient({ + subscribeEvents: vi.fn(() => + envelopeStream([envelope('turn_error', { message: 'boom' })]), + ), + }); + const adaptor = makeAdaptor(client); + + const events = await collect(adaptor, handleFor()); + + expect(events).toEqual([{ type: 'turn_error', error: 'boom' }]); + }); + + it('maps prompt_cancelled to a cancelled turn_error', async () => { + const client = makeClient({ + subscribeEvents: vi.fn(() => + envelopeStream([envelope('prompt_cancelled', {}, { promptId: 'p1' })]), + ), + }); + const adaptor = makeAdaptor(client); + const handle = handleFor(); + + const events = await collect(adaptor, handle); + + expect(events).toEqual([ + { type: 'turn_error', jobRef: 'p1', error: 'cancelled' }, + ]); + expect(adaptor.isBusy(handle)).toBe(false); + }); + + it('normalizes permission_request with a descriptive title and classified options', async () => { + const client = makeClient({ + subscribeEvents: vi.fn(() => + envelopeStream([permissionRequestEnvelope('req-1')]), + ), + }); + const adaptor = makeAdaptor(client); + + const events = await collect(adaptor, handleFor()); + + expect(events).toHaveLength(1); + const event = events[0]!; + expect(event.type).toBe('permission_request'); + if (event.type !== 'permission_request') return; + expect(event.requestId).toBe('req-1'); + expect(event.title).toContain('Bash'); + expect(event.title).toContain('rm -rf /tmp'); + expect(event.options).toEqual([ + { + optionId: 'allow', + label: 'Allow once', + kind: 'proceed', + escalation: 'once', + }, + { optionId: 'deny', kind: 'reject' }, + ]); + }); + + it('attributes permission_resolved by comparing the originator clientId', async () => { + const client = makeClient({ + subscribeEvents: vi.fn(() => + envelopeStream([ + envelope( + 'permission_resolved', + { requestId: 'req-ours' }, + { originatorClientId: ISSUED_CLIENT_ID }, + ), + envelope( + 'permission_resolved', + { requestId: 'req-theirs' }, + { originatorClientId: 'webshell-42' }, + ), + ]), + ), + }); + const adaptor = makeAdaptor(client); + // Register the session so the daemon-issued clientId is tracked. + await adaptor.createSession(); + + const events = await collect(adaptor, handleFor()); + + expect(events).toEqual([ + { type: 'permission_resolved', requestId: 'req-ours', byUs: true }, + { type: 'permission_resolved', requestId: 'req-theirs', byUs: false }, + ]); + }); + + it('strips terminal escapes and keeps only the first line of tool titles', async () => { + const client = makeClient({ + subscribeEvents: vi.fn(() => + envelopeStream([ + envelope('session_update', { + update: { + sessionUpdate: 'tool_call', + title: '\x1b[31mRun\x1b[0m npm\ntest', + }, + }), + ]), + ), + }); + const adaptor = makeAdaptor(client); + + const events = await collect(adaptor, handleFor()); + + expect(events).toEqual([{ type: 'progress', summary: 'Run npm' }]); + }); + + it('never starts a clamped summary with a lone low surrogate', async () => { + // 5001 UTF-16 units; the 4000-unit tail cut lands between the halves of + // an emoji surrogate pair unless the clamp snaps off it. + const detail = `${'😀'.repeat(1000)}x${'😀'.repeat(1500)}`; + const client = makeClient({ + subscribeEvents: vi.fn(() => + envelopeStream([ + envelope('session_update', { + update: { + sessionUpdate: 'agent_message_chunk', + content: { type: 'text', text: detail }, + }, + }), + envelope('turn_complete', {}, { promptId: 'p1' }), + ]), + ), + }); + const adaptor = makeAdaptor(client); + + const events = await collect(adaptor, handleFor()); + + const complete = events.find((event) => event.type === 'turn_complete'); + if (complete?.type !== 'turn_complete') throw new Error('no turn_complete'); + expect(complete.summary.startsWith('…')).toBe(true); + const lead = complete.summary.charCodeAt(1); + expect(lead >= 0xdc00 && lead <= 0xdfff).toBe(false); + expect(complete.summary.length).toBe(4000); // '…' + 3999 snapped units + }); + + it('keeps the turn-buffer cap off surrogate boundaries', async () => { + // 48001 units: the 48000-unit buffer cap cut lands mid-pair. + const oversized = `${'😀'.repeat(24000)}x`; + const client = makeClient({ + subscribeEvents: vi.fn(() => + envelopeStream([ + envelope('session_update', { + update: { + sessionUpdate: 'agent_message_chunk', + content: { type: 'text', text: oversized }, + }, + }), + envelope('turn_complete', {}, { promptId: 'p1' }), + ]), + ), + }); + const adaptor = makeAdaptor(client); + + const events = await collect(adaptor, handleFor()); + + const complete = events.find((event) => event.type === 'turn_complete'); + if (complete?.type !== 'turn_complete') throw new Error('no turn_complete'); + const lead = complete.detail?.charCodeAt(0) ?? 0; + expect(lead >= 0xdc00 && lead <= 0xdfff).toBe(false); + }); + + it('attributes byUs via the own-vote record for adopted sessions', async () => { + // Adopted via session_list: no createSession, so no daemon-issued + // clientId and no originator stamp on the anonymous vote. + const client = makeClient(); + const adaptor = makeAdaptor(client); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([permissionRequestEnvelope('req-1')]), + ); + await collect(adaptor, handleFor()); + + await adaptor.respondPermission(handleFor(), 'req-1', 'allow'); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([envelope('permission_resolved', { requestId: 'req-1' })]), + ); + const events = await collect(adaptor, handleFor()); + + expect(events).toEqual([ + { type: 'permission_resolved', requestId: 'req-1', byUs: true }, + ]); + }); + + it('does not claim byUs for a vote the daemon refused', async () => { + const client = makeClient({ + respondToSessionPermission: vi.fn(async () => false), + }); + const adaptor = makeAdaptor(client); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([permissionRequestEnvelope('req-1')]), + ); + await collect(adaptor, handleFor()); + + await adaptor.respondPermission(handleFor(), 'req-1', 'allow'); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([envelope('permission_resolved', { requestId: 'req-1' })]), + ); + const events = await collect(adaptor, handleFor()); + + expect(events).toEqual([ + { type: 'permission_resolved', requestId: 'req-1', byUs: false }, + ]); + }); + + it('drops stored permission options when the request resolves elsewhere', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + await seedPermissionOptions(adaptor, client); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([ + envelope( + 'permission_resolved', + { requestId: 'req-1' }, + { originatorClientId: 'webshell-42' }, + ), + ]), + ); + await collect(adaptor, handleFor()); + + // With the stored options gone, a late vote can only fall back to the + // cancelled outcome instead of replaying a stale optionId. + await adaptor.respondPermission(handleFor(), 'req-1', 'allow'); + + expect(client.respondToSessionPermission).toHaveBeenCalledWith( + SESSION_ID, + 'req-1', + { outcome: { outcome: 'cancelled' } }, + ISSUED_CLIENT_ID, + ); + }); + + it('emits session_closed and stops consuming the upstream generator', async () => { + let pulls = 0; + let finalized = false; + async function* stream(): EnvelopeStream { + try { + pulls += 1; + yield envelope('session_closed'); + pulls += 1; + yield envelope('turn_error', { message: 'never delivered' }); + } finally { + finalized = true; + } + } + const client = makeClient({ subscribeEvents: vi.fn(() => stream()) }); + const adaptor = makeAdaptor(client); + const handle = handleFor(); + + const events = await collect(adaptor, handle); + + expect(events).toEqual([{ type: 'session_closed' }]); + expect(pulls).toBe(1); + expect(finalized).toBe(true); + expect(adaptor.isBusy(handle)).toBe(false); + }); + + it('ignores unknown envelope types', async () => { + const client = makeClient({ + subscribeEvents: vi.fn(() => + envelopeStream([ + envelope('git_status_changed', { branch: 'main' }), + envelope('some_future_thing'), + ]), + ), + }); + const adaptor = makeAdaptor(client); + + const events = await collect(adaptor, handleFor()); + + expect(events).toEqual([]); + }); +}); + +describe('QwenCodeAdaptor.respondPermission', () => { + it('votes the proceed option for an allow decision', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + await seedPermissionOptions(adaptor, client); + + const result = await adaptor.respondPermission( + handleFor(), + 'req-1', + 'allow', + ); + + expect(client.respondToSessionPermission).toHaveBeenCalledWith( + SESSION_ID, + 'req-1', + { outcome: { outcome: 'selected', optionId: 'allow' } }, + ISSUED_CLIENT_ID, + ); + expect(result).toBe('delivered'); + }); + + it('classifies real serve snake_case option ids (proceed_once)', async () => { + // Regression: `\b` treats "_" as a word character, so a boundary match + // against the raw id never fires — an allow vote silently degraded to a + // cancelled outcome and serve dropped the turn. + const client = makeClient(); + const adaptor = makeAdaptor(client); + await adaptor.createSession(); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([ + envelope('permission_request', { + requestId: 'req-1', + toolCall: { name: 'write_file' }, + // Bare ids, no labels: nothing else may rescue the classification. + options: [ + { optionId: 'proceed_once' }, + { optionId: 'proceed_always' }, + { optionId: 'cancel' }, + ], + }), + ]), + ); + await collect(adaptor, handleFor()); + + const result = await adaptor.respondPermission( + handleFor(), + 'req-1', + 'allow', + ); + + expect(client.respondToSessionPermission).toHaveBeenCalledWith( + SESSION_ID, + 'req-1', + { outcome: { outcome: 'selected', optionId: 'proceed_once' } }, + ISSUED_CLIENT_ID, + ); + expect(result).toBe('delivered'); + }); + + it('votes the least-escalating allow option in production order', async () => { + // qwen serve offers always-allow options FIRST (toPermissionOptions); + // a bare voice "allow" must still take the one-shot grant, not persist + // a project-scoped always-allow rule. + const client = makeClient(); + const adaptor = makeAdaptor(client); + await adaptor.createSession(); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([ + envelope('permission_request', { + requestId: 'req-1', + toolCall: { name: 'Bash', command: 'git push' }, + options: [ + { + optionId: 'proceed_always_project', + name: 'Always Allow in project: git', + kind: 'allow_always', + }, + { + optionId: 'proceed_always_user', + name: 'Always Allow for user: git', + kind: 'allow_always', + }, + { optionId: 'proceed_once', name: 'Allow', kind: 'allow_once' }, + { optionId: 'cancel', name: 'Reject', kind: 'reject_once' }, + ], + }), + ]), + ); + await collect(adaptor, handleFor()); + + const result = await adaptor.respondPermission( + handleFor(), + 'req-1', + 'allow', + ); + + expect(client.respondToSessionPermission).toHaveBeenCalledWith( + SESSION_ID, + 'req-1', + { outcome: { outcome: 'selected', optionId: 'proceed_once' } }, + ISSUED_CLIENT_ID, + ); + expect(result).toBe('delivered'); + }); + + it('votes the least-escalating reject option for a deny decision', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + await adaptor.createSession(); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([ + envelope('permission_request', { + requestId: 'req-1', + toolCall: { name: 'Bash', command: 'git push' }, + options: [ + { + optionId: 'reject_always', + name: 'Always Reject', + kind: 'reject_always', + }, + { optionId: 'cancel', name: 'Reject', kind: 'reject_once' }, + ], + }), + ]), + ); + await collect(adaptor, handleFor()); + + await adaptor.respondPermission(handleFor(), 'req-1', 'deny'); + + expect(client.respondToSessionPermission).toHaveBeenCalledWith( + SESSION_ID, + 'req-1', + { outcome: { outcome: 'selected', optionId: 'cancel' } }, + ISSUED_CLIENT_ID, + ); + }); + + it('fails closed to cancelled when the wire kinds are unknown', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + await adaptor.createSession(); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([ + envelope('permission_request', { + requestId: 'req-1', + toolCall: { name: 'write_file' }, + // A structured kind we do not understand must not be votable + // through the word heuristics. + options: [{ optionId: 'proceed_once', kind: 'allow_v2_mystery' }], + }), + ]), + ); + await collect(adaptor, handleFor()); + + await adaptor.respondPermission(handleFor(), 'req-1', 'allow'); + + expect(client.respondToSessionPermission).toHaveBeenCalledWith( + SESSION_ID, + 'req-1', + { outcome: { outcome: 'cancelled' } }, + ISSUED_CLIENT_ID, + ); + }); + + it('votes the reject option for a deny decision', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + await seedPermissionOptions(adaptor, client); + + const result = await adaptor.respondPermission( + handleFor(), + 'req-1', + 'deny', + ); + + expect(client.respondToSessionPermission).toHaveBeenCalledWith( + SESSION_ID, + 'req-1', + { outcome: { outcome: 'selected', optionId: 'deny' } }, + ISSUED_CLIENT_ID, + ); + expect(result).toBe('delivered'); + }); + + it('votes cancelled for a cancel decision', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + await seedPermissionOptions(adaptor, client); + + await adaptor.respondPermission(handleFor(), 'req-1', 'cancel'); + + expect(client.respondToSessionPermission).toHaveBeenCalledWith( + SESSION_ID, + 'req-1', + { outcome: { outcome: 'cancelled' } }, + ISSUED_CLIENT_ID, + ); + }); + + it('reports already_resolved when the daemon refuses the vote', async () => { + const client = makeClient({ + respondToSessionPermission: vi.fn(async () => false), + }); + const adaptor = makeAdaptor(client); + await seedPermissionOptions(adaptor, client); + + const result = await adaptor.respondPermission( + handleFor(), + 'req-1', + 'allow', + ); + + expect(result).toBe('already_resolved'); + }); + + it('falls back to cancelled when no options are known for the request', async () => { + const client = makeClient(); + const adaptor = makeAdaptor(client); + await adaptor.createSession(); + + await adaptor.respondPermission(handleFor(), 'req-unknown', 'allow'); + + expect(client.respondToSessionPermission).toHaveBeenCalledWith( + SESSION_ID, + 'req-unknown', + { outcome: { outcome: 'cancelled' } }, + ISSUED_CLIENT_ID, + ); + }); +}); + +describe('QwenCodeAdaptor.close', () => { + it('clears all tracked session state', async () => { + const client = makeClient({ + createOrAttachSession: vi.fn(async () => ({ + sessionId: SESSION_ID, + hasActivePrompt: true, + clientId: ISSUED_CLIENT_ID, + })), + }); + const adaptor = makeAdaptor(client); + const handle = await adaptor.createSession(); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([permissionRequestEnvelope('req-1')]), + ); + await collect(adaptor, handle); + expect(adaptor.isBusy(handle)).toBe(true); + + await adaptor.close(); + + // Busy flag, issued clientId, and stored permission options are gone: + // isBusy resets and a late vote can only fall back to the anonymous + // cancelled outcome. + expect(adaptor.isBusy(handle)).toBe(false); + await adaptor.respondPermission(handle, 'req-1', 'allow'); + expect(client.respondToSessionPermission).toHaveBeenCalledWith( + SESSION_ID, + 'req-1', + { outcome: { outcome: 'cancelled' } }, + undefined, + ); + }); +}); + +describe('QwenCodeAdaptor prompt image blocks', () => { + it('uploads image blocks and splices the returned reference into the prompt', async () => { + const reference = { type: 'resource_link', uri: 'attachment://shot' }; + const client = makeClient({ + uploadSessionAttachment: vi.fn(async () => reference), + }); + const adaptor = makeAdaptor(client); + const handle = await adaptor.createSession(); + const blocks: ContentBlock[] = [ + { type: 'text', text: 'see the screenshot' }, + { + type: 'image', + mimeType: 'image/png', + data: new Uint8Array([1, 2, 3]), + name: 'shot.png', + }, + ]; + + await adaptor.prompt(handle, blocks); + + const uploadCall = vi.mocked(client.uploadSessionAttachment).mock.calls[0]!; + expect(uploadCall[0]).toBe(SESSION_ID); + expect(uploadCall[1]).toBeInstanceOf(Blob); + expect(uploadCall[1].type).toBe('image/png'); + expect(uploadCall[1].size).toBe(3); + expect(uploadCall[2]).toBe('shot.png'); + expect(uploadCall[3]).toBe('image/png'); + + const promptCall = vi.mocked(client.promptNonBlocking).mock.calls[0]!; + const body = promptCall[1] as { prompt: unknown[] }; + expect(body.prompt).toHaveLength(2); + expect(body.prompt[0]).toEqual({ + type: 'text', + text: 'see the screenshot', + }); + expect(body.prompt[1]).toBe(reference); + }); + + it('attributes a resolution to the daemon stamp, not our own in-flight vote', async () => { + // R1-2 regression pin: we record the vote BEFORE the HTTP round-trip + // (the SSE resolution can beat the response). When another client + // resolves the request during that window, the frame carries THEIR + // originatorClientId — byUs must be false even though ownVotes still + // holds the requestId. + const client = makeClient(); + const adaptor = makeAdaptor(client); + await adaptor.createSession(); + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([permissionRequestEnvelope('req-1')]), + ); + await collect(adaptor, handleFor()); + + // Park the vote mid-round-trip, then let the pump deliver the + // resolution stamped with the OTHER client's id. + let releaseVote: (() => void) | undefined; + vi.mocked(client.respondToSessionPermission).mockImplementation( + () => + new Promise((resolve) => { + releaseVote = () => resolve(true); + }), + ); + const vote = adaptor.respondPermission(handleFor(), 'req-1', 'allow'); + await vi.waitFor(() => { + expect(client.respondToSessionPermission).toHaveBeenCalled(); + }); + + // Feed the resolution through a second stream subscription. + vi.mocked(client.subscribeEvents).mockReturnValueOnce( + envelopeStream([ + envelope( + 'permission_resolved', + { requestId: 'req-1' }, + { originatorClientId: 'webshell-42' }, + ), + ]), + ); + const resolved = await collect(adaptor, handleFor()); + releaseVote?.(); + await vote; + + expect( + resolved.some( + (event) => event.type === 'permission_resolved' && event.byUs, + ), + ).toBe(false); + }); +}); diff --git a/packages/qwen-live/src/adaptor/qwen-code-adaptor.ts b/packages/qwen-live/src/adaptor/qwen-code-adaptor.ts new file mode 100644 index 00000000000..7a6ce64a80f --- /dev/null +++ b/packages/qwen-live/src/adaptor/qwen-code-adaptor.ts @@ -0,0 +1,837 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * BackendAdaptor implementation that drives a running `qwen serve` daemon + * over its REST/SSE surface via the official `@qwen-code/sdk` DaemonClient. + * + * Notes anchored to daemon behavior this adaptor relies on: + * - `POST /session/:id/prompt` answers 202 with a `promptId` — that is the + * admission receipt the orchestrator's handoff tool returns on. + * - `agent_message_chunk` / `tool_call` live inside + * `session_update.data.update.sessionUpdate`, not as top-level types. + * - `POST /session/:id/mid-turn-message` hardcodes reject-if-idle, which is + * exactly the probe `prompt({steer:true})` needs: accepted means the + * instruction joined the running turn, rejected means the session went + * idle and a normal prompt is the right fallback. + * - `permission_resolved.originatorClientId` names the *voter*, so comparing + * it against the daemon-issued clientId for the session distinguishes "we + * answered" from "someone answered in WebShell". The issued id comes back + * on the create/attach response; self-made ids are rejected by the + * daemon's client registration guard, so every per-session call echoes + * the issued one. Sessions adopted via `session_list` never receive an + * issued id, so the adaptor also records every requestId it voted on and + * attributes a resolution for one of those to itself. + */ + +import { DaemonClient } from '@qwen-code/sdk'; +import type { + BackendAdaptor, + BackendCapabilities, + BackendEvent, + BackendHandle, + ContentBlock, + PermissionDecision, + PermissionOption, + PermissionOptionKind, + PromptReceipt, + SessionSummary, +} from './types.js'; + +const ADAPTOR_NAME = 'qwen-code'; + +/** Feature tags this adaptor refuses to run without (all `since: v1`). */ +const REQUIRED_FEATURES = [ + 'session_create', + 'session_prompt', + 'session_events', + 'session_cancel', + 'session_permission_vote', + 'session_mid_turn_message_mutation', +] as const; + +/** Keep receipts and summaries bounded; backends can produce huge turns. */ +const MAX_SUMMARY_CHARS = 4_000; +const MAX_DETAIL_CHARS = 48_000; + +/** + * Structural subset of DaemonClient used by this adaptor. Unit tests inject + * a fake; production passes a real DaemonClient (or omits it to have one + * constructed from `baseUrl`/`token`). + */ +export interface DaemonClientLike { + capabilities(): Promise<{ + features?: readonly string[]; + workspaceCwd?: string; + }>; + createOrAttachSession( + req: Record, + clientId?: string, + ): Promise<{ + sessionId: string; + hasActivePrompt?: boolean; + clientId?: string; + }>; + listWorkspaceSessions( + workspaceCwd: string, + options?: Record, + ): Promise>>; + promptNonBlocking( + sessionId: string, + req: Record, + signal?: AbortSignal, + clientId?: string, + ): Promise>; + subscribeEvents( + sessionId: string, + opts?: Record, + ): AsyncGenerator<{ + id?: number; + v: 1; + type: string; + data: unknown; + promptId?: string; + originatorClientId?: string; + }>; + enqueueMidTurnMessage( + sessionId: string, + message: string, + opts?: Record, + ): Promise<{ accepted: boolean; messageId?: string }>; + cancel(sessionId: string, clientId?: string): Promise; + respondToSessionPermission( + sessionId: string, + requestId: string, + response: Record, + clientId?: string, + ): Promise; + uploadSessionAttachment( + sessionId: string, + data: Blob, + name: string, + mimeType: string, + opts?: Record, + ): Promise>; + updateSessionMetadata( + sessionId: string, + metadata: { displayName?: string }, + clientId?: string, + ): Promise<{ displayName?: string }>; +} + +export interface QwenCodeAdaptorOptions { + /** qwen serve base URL, e.g. `http://127.0.0.1:4170`. */ + baseUrl: string; + token?: string; + /** Default cwd for new sessions; falls back to the daemon's workspaceCwd. */ + defaultCwd?: string; + /** Stable client identity used for permission-vote attribution. */ + clientId?: string; + /** Injection seam for unit tests. */ + client?: DaemonClientLike; +} + +interface SessionState { + busy: boolean; + activeJobRef?: string; + /** Daemon-issued client id for this session; echoed on every call. */ + clientId?: string; + /** Accumulated assistant text for the current turn. */ + turnBuffer: string; + /** Options captured per pending permission request, for decision mapping. */ + permissionOptions: Map; + /** + * Permission requestIds this adaptor voted on. Sessions adopted via + * session_list carry no daemon-issued clientId, so originator comparison + * alone can never attribute our own vote — this set is the fallback. + * Entries are removed when the request resolves. + */ + ownVotes: Set; + closed: boolean; +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +/** + * Map one wire permission option to the adaptor's kind + escalation. + * + * qwen serve stamps every option with the ACP structured `kind` + * (`allow_once` | `allow_always` | `reject_once` | `reject_always` — see + * `toPermissionOptions` in packages/cli acp-integration/permissionUtils.ts); + * that is authoritative when present. An unknown structured kind fails + * closed to 'other'. The word heuristics run only when the backend omitted + * the field entirely. + */ +function classifyOption( + optionId: string, + name: string | undefined, + wireKind: string | undefined, +): { kind: PermissionOptionKind; escalation?: 'once' | 'always' } { + switch (wireKind) { + case 'allow_once': + return { kind: 'proceed', escalation: 'once' }; + case 'allow_always': + return { kind: 'proceed', escalation: 'always' }; + case 'reject_once': + return { kind: 'reject', escalation: 'once' }; + case 'reject_always': + return { kind: 'reject', escalation: 'always' }; + default: + break; + } + // Fail closed: a structured kind we do not understand must not be votable + // through a bare "allow"/"deny". + if (wireKind !== undefined) return { kind: 'other' }; + // Fallback for backends that omit the structured kind. Snake/kebab ids + // ("proceed_once") must split into words — `\b` treats an underscore as a + // word character and would never match inside them. + const haystack = `${optionId} ${name ?? ''}` + .toLowerCase() + .replace(/[_-]/g, ' '); + const escalation = /\balways\b/.test(haystack) + ? ('always' as const) + : /\bonce\b/.test(haystack) + ? ('once' as const) + : undefined; + // Word boundaries matter: a bare /no/ would match inside "notify" and + // misroute an allow vote to a reject option. + if (/\b(allow|proceed|approve|yes|accept)\b/.test(haystack)) { + return { kind: 'proceed', ...(escalation ? { escalation } : {}) }; + } + if (/\b(deny|reject|refuse|no|cancel)\b/.test(haystack)) { + return { kind: 'reject', ...(escalation ? { escalation } : {}) }; + } + return { kind: 'other' }; +} + +/** + * The narrowest option of the wanted kind. A bare voice "allow" must take + * the one-shot grant, never persist an always-allow rule (serve offers + * [proceed_always_project, proceed_always_user, proceed_once, cancel] — + * first-match would pick the project-wide rule). + */ +function pickLeastEscalating( + options: readonly PermissionOption[], + wanted: PermissionOptionKind, +): PermissionOption | undefined { + const rank = (option: PermissionOption): number => + option.escalation === 'once' ? 0 : option.escalation === undefined ? 1 : 2; + let best: PermissionOption | undefined; + for (const candidate of options) { + if (candidate.kind !== wanted) continue; + if (best === undefined || rank(candidate) < rank(best)) best = candidate; + } + return best; +} + +/** + * Compose the human-readable permission title. Control sequences are + * stripped (the title flows verbatim into the spoken ask and keys the + * broker's standing rule — raw ESC/OSC bytes must not reach speech or + * anchor a trust grant), but NOT truncated to the first line: the + * standing-rule key must span the whole command. + */ +function describeToolCall(toolCall: unknown): string { + if (!isRecord(toolCall)) return 'a tool call'; + const name = typeof toolCall['name'] === 'string' ? toolCall['name'] : ''; + const command = + typeof toolCall['command'] === 'string' ? toolCall['command'] : ''; + const title = typeof toolCall['title'] === 'string' ? toolCall['title'] : ''; + const detail = stripControlSequences(command || title); + if (name && detail) return `${name}: ${detail}`; + return name || detail || 'a tool call'; +} + +/** + * Remove terminal control sequences without the first-line cut that + * sanitizeTitleLine applies — the permission key needs the whole command. + * Mirrors the same sequence families (OSC, CSI, SS2/SS3/DCS, C0/DEL/C1). + */ +function stripControlSequences(text: string): string { + return ( + text + // eslint-disable-next-line no-control-regex + .replace(/\x1b\][\s\S]*?(?:\x07|\x1b\\)/g, '') + // eslint-disable-next-line no-control-regex + .replace(/\x1b\[[0-?]*[ -/]*[@-~]/g, '') + // eslint-disable-next-line no-control-regex + .replace(/\x1b[NOP]/g, '') + // eslint-disable-next-line no-control-regex + .replace(/[\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\x9f]/g, ' ') + ); +} + +/** + * Slice the last `max` UTF-16 units, snapping the cut off a split surrogate + * pair: a cut landing between the halves would lead with a lone low + * surrogate that renders/speaks as U+FFFD. + */ +function tailSlice(text: string, max: number): string { + let start = text.length - max; + const unit = text.charCodeAt(start); + if (unit >= 0xdc00 && unit <= 0xdfff) start += 1; + return text.slice(start); +} + +function clampTail(text: string, max: number): string { + const trimmed = text.trim(); + if (trimmed.length <= max) return trimmed; + return `…${tailSlice(trimmed, max)}`; +} + +/** + * One clean line for a tool-call title: first line only, terminal control + * sequences stripped. Minimal inline of core's stripTerminalControlSequences + * (packages/core utils/terminalSafe.ts) — the monolith's voice consumer + * applied the same guard before titles reached the realtime model + * (live-session-coordinator.ts), and neither acp-bridge nor the daemon + * sanitizes upstream. + */ +function sanitizeTitleLine(title: string): string { + const firstLine = title.split(/\r?\n/, 1)[0] ?? ''; + return ( + firstLine + // OSC: ESC ] ... (BEL | ST) + // eslint-disable-next-line no-control-regex + .replace(/\x1b\][\s\S]*?(?:\x07|\x1b\\)/g, '') + // CSI: ESC [ params intermediates final + // eslint-disable-next-line no-control-regex + .replace(/\x1b\[[0-?]*[ -/]*[@-~]/g, '') + // SS2/SS3/DCS leaders + // eslint-disable-next-line no-control-regex + .replace(/\x1b[NOP]/g, '') + // Remaining C0 controls + DEL + C1 controls + // eslint-disable-next-line no-control-regex + .replace(/[\x00-\x1f\x7f-\x9f]/g, ' ') + .trim() + ); +} + +export class QwenCodeAdaptor implements BackendAdaptor { + readonly name = ADAPTOR_NAME; + + private readonly client: DaemonClientLike; + private readonly options: QwenCodeAdaptorOptions; + private readonly sessions = new Map(); + /** Every cwd sessions were created in; listSessions unions across them. */ + private readonly sessionCwds = new Set(); + private workspaceCwd: string | undefined; + + constructor(options: QwenCodeAdaptorOptions) { + this.options = options; + this.client = + options.client ?? + (new DaemonClient({ + baseUrl: options.baseUrl, + ...(options.token ? { token: options.token } : {}), + }) as unknown as DaemonClientLike); + } + + capabilities(): BackendCapabilities { + return { + steering: 'native', + imageInput: true, + permissionForwarding: true, + // Session-to-session messaging (proactive speak) is milestone M3. + proactiveSpeak: false, + sessionList: true, + eventDelivery: 'stream', + }; + } + + async preflight(): Promise { + let caps: Awaited>; + try { + caps = await this.client.capabilities(); + } catch (error) { + const detail = error instanceof Error ? error.message : String(error); + const status = + isRecord(error) && typeof error['status'] === 'number' + ? error['status'] + : undefined; + if (status !== undefined) { + // The daemon answered — it is running but refused the request. + // A "start it" hint would send the operator hunting for a daemon + // that already exists. + throw new Error( + `qwen serve at ${this.options.baseUrl} responded with HTTP ` + + `${status} and refused the request: ${detail}. ` + + 'Check the auth token and base URL.', + ); + } + throw new Error( + `qwen serve is not reachable at ${this.options.baseUrl}: ` + + `${detail}. ` + + 'Start it with `qwen serve` before launching qwen-live.', + ); + } + const features = new Set(caps.features ?? []); + const missing = REQUIRED_FEATURES.filter((f) => !features.has(f)); + if (missing.length > 0) { + throw new Error( + `qwen serve at ${this.options.baseUrl} is missing required ` + + `capabilities: ${missing.join(', ')}. Upgrade qwen-code.`, + ); + } + this.workspaceCwd = caps.workspaceCwd; + } + + async createSession(opts?: { + cwd?: string; + label?: string; + }): Promise { + const cwd = opts?.cwd ?? this.options.defaultCwd ?? this.workspaceCwd; + const session = await this.client.createOrAttachSession( + { + ...(cwd !== undefined ? { workspaceCwd: cwd } : {}), + sessionScope: 'thread', + sourceType: 'qwen-live', + }, + this.options.clientId, + ); + if (cwd !== undefined) this.sessionCwds.add(cwd); + const state = this.trackSession(session.sessionId, { + busy: session.hasActivePrompt === true, + // The daemon issues the authoritative per-client id on create/attach; + // every later call for this session must echo it (a self-made id is + // rejected by the daemon's client registration guard). Older daemons + // omit it — those calls stay anonymous. + ...(typeof session.clientId === 'string' + ? { clientId: session.clientId } + : {}), + }); + if (opts?.label !== undefined) { + // The daemon session's displayName is what listSessions (here and in + // every other client) reports back — the requested label is useless + // unless it is applied to the daemon session itself. + await this.client.updateSessionMetadata( + session.sessionId, + { displayName: opts.label }, + state.clientId, + ); + } + return { id: session.sessionId, adaptor: ADAPTOR_NAME }; + } + + async listSessions(): Promise { + const defaultCwd = this.options.defaultCwd ?? this.workspaceCwd; + const cwds = new Set(); + if (defaultCwd !== undefined) cwds.add(defaultCwd); + for (const cwd of this.sessionCwds) cwds.add(cwd); + const summaries = new Map(); + for (const cwd of cwds) { + // The daemon catalog accumulates sessions from every client and the + // SDK defaults to a 20-row first page; 1000 is the server-side clamp. + const sessions = await this.client.listWorkspaceSessions(cwd, { + pageSize: 1000, + }); + for (const raw of sessions) { + const sessionId = raw['sessionId']; + if (typeof sessionId !== 'string' || summaries.has(sessionId)) { + continue; + } + const tracked = this.sessions.get(sessionId); + const label = raw['displayName']; + const hasActivePrompt = raw['hasActivePrompt']; + summaries.set(sessionId, { + handle: { id: sessionId, adaptor: ADAPTOR_NAME }, + ...(typeof label === 'string' ? { label } : {}), + cwd, + state: tracked + ? tracked.closed + ? ('closed' as const) + : tracked.busy + ? ('busy' as const) + : ('idle' as const) + : // Untracked (e.g. after a qwen-live restart): the wire record + // itself says whether a prompt is running. + typeof hasActivePrompt === 'boolean' + ? hasActivePrompt + ? ('busy' as const) + : ('idle' as const) + : ('unknown' as const), + }); + } + } + return [...summaries.values()]; + } + + async prompt( + handle: BackendHandle, + blocks: readonly ContentBlock[], + opts?: { steer?: boolean }, + ): Promise { + const state = this.trackSession(handle.id); + + if (opts?.steer && state.busy) { + // Mid-turn injection is text-only on the wire; a handoff carrying + // image attachments must go through a full prompt so the images are + // not silently dropped — the fall-through below queues it as the + // session's next turn. + const hasImages = blocks.some((block) => block.type === 'image'); + if (!hasImages) { + const message = blocks + .filter( + (block): block is Extract => + block.type === 'text', + ) + .map((block) => block.text) + .join('\n\n'); + let steered: { accepted: boolean; messageId?: string } | undefined; + try { + steered = await this.client.enqueueMidTurnMessage( + handle.id, + message, + state.clientId !== undefined ? { clientId: state.clientId } : {}, + ); + } catch (error) { + // The mid-turn route 400s on over-long messages; the identical + // payload is admissible as a queued full prompt, so fall through + // instead of failing the handoff. Anything else is a real error. + if (!(isRecord(error) && error['status'] === 400)) throw error; + } + if (steered?.accepted) { + return { + status: 'accepted', + joinedActiveTurn: true, + ...(state.activeJobRef !== undefined + ? { jobRef: state.activeJobRef } + : {}), + note: 'joined the currently running task', + }; + } + } + // Either the turn ended between our busy check and the injection, or + // the payload needs a full prompt: fall through — that is exactly the + // "queued as the next turn" tier. + } + + const prompt = await this.buildPromptBlocks(handle.id, blocks); + let accepted: Record; + try { + accepted = await this.client.promptNonBlocking( + handle.id, + { prompt }, + undefined, + state.clientId, + ); + } catch (error) { + // Queue-full comes in two shapes: DaemonHttpError carries a top-level + // 503 status; the SDK's client-side guard throws + // DaemonPendingPromptLimitError (no status field). + if ( + isRecord(error) && + (error['status'] === 503 || + error['name'] === 'DaemonPendingPromptLimitError') + ) { + return { + status: 'rejected', + note: 'the session is busy and its queue is full; wait or stop the current task first', + }; + } + throw error; + } + const promptId = accepted['promptId']; + const jobRef = typeof promptId === 'string' ? promptId : undefined; + // Only advance activeJobRef when this prompt actually starts the + // session's turn: a busy-session fallback (image steer, over-long text + // 400) queues behind the running turn, and overwriting the ref here + // would make a later accepted steer report the QUEUED prompt's id as + // the running turn's jobRef. pending_prompt_started establishes it + // when the queued prompt eventually runs. + const wasBusy = state.busy; + state.busy = true; + if (!wasBusy && jobRef !== undefined) state.activeJobRef = jobRef; + return { + status: opts?.steer ? 'queued' : 'accepted', + ...(jobRef !== undefined ? { jobRef } : {}), + ...(opts?.steer + ? { note: 'queued as the next task in that session' } + : {}), + }; + } + + async *events( + handle: BackendHandle, + opts?: { signal?: AbortSignal }, + ): AsyncIterable { + const state = this.trackSession(handle.id); + const stream = this.client.subscribeEvents(handle.id, { + ...(opts?.signal ? { signal: opts.signal } : {}), + ...(state.clientId !== undefined ? { clientId: state.clientId } : {}), + }); + for await (const envelope of stream) { + const events = this.normalize(state, envelope); + for (const event of events) { + yield event; + if (event.type === 'session_closed') return; + } + } + } + + isBusy(handle: BackendHandle): boolean { + return this.sessions.get(handle.id)?.busy === true; + } + + async cancel(handle: BackendHandle): Promise { + await this.client.cancel(handle.id, this.sessions.get(handle.id)?.clientId); + } + + async respondPermission( + handle: BackendHandle, + requestId: string, + decision: PermissionDecision, + ): Promise<'delivered' | 'already_resolved'> { + const state = this.trackSession(handle.id); + const options = state.permissionOptions.get(requestId) ?? []; + let response: Record; + if (decision === 'cancel') { + response = { outcome: { outcome: 'cancelled' } }; + } else { + const wanted: PermissionOptionKind = + decision === 'allow' ? 'proceed' : 'reject'; + const option = pickLeastEscalating(options, wanted); + response = option + ? { outcome: { outcome: 'selected', optionId: option.optionId } } + : { outcome: { outcome: 'cancelled' } }; + } + // Record the vote BEFORE the HTTP round-trip: the daemon publishes the + // SSE permission_resolved before the vote response returns, so waiting + // for `delivered` can lose the race against our own event pump — and on + // adopted sessions the own-vote record is the only byUs signal. + state.ownVotes.add(requestId); + let delivered: boolean; + try { + delivered = await this.client.respondToSessionPermission( + handle.id, + requestId, + response, + state.clientId, + ); + } catch (error) { + state.ownVotes.delete(requestId); + throw error; + } + if (delivered) { + state.permissionOptions.delete(requestId); + } else { + // Someone else settled it first; our vote did not count. + state.ownVotes.delete(requestId); + } + return delivered ? 'delivered' : 'already_resolved'; + } + + async close(): Promise { + this.sessions.clear(); + } + + // -- internals ----------------------------------------------------------- + + private trackSession( + sessionId: string, + seed?: Partial>, + ): SessionState { + let state = this.sessions.get(sessionId); + if (!state) { + state = { + busy: seed?.busy ?? false, + turnBuffer: '', + permissionOptions: new Map(), + ownVotes: new Set(), + closed: false, + }; + this.sessions.set(sessionId, state); + } else if (seed?.busy !== undefined) { + state.busy = seed.busy; + } + if (seed?.clientId !== undefined) state.clientId = seed.clientId; + return state; + } + + private async buildPromptBlocks( + sessionId: string, + blocks: readonly ContentBlock[], + ): Promise>> { + const prompt: Array> = []; + for (const block of blocks) { + if (block.type === 'text') { + prompt.push({ type: 'text', text: block.text }); + continue; + } + // Image: upload as a session attachment and reference it, so the + // prompt body stays small and the daemon owns the bytes. + const reference = await this.client.uploadSessionAttachment( + sessionId, + new Blob([block.data as BlobPart], { type: block.mimeType }), + block.name ?? 'attachment', + block.mimeType, + ); + prompt.push(reference); + } + return prompt; + } + + private normalize( + state: SessionState, + envelope: { + type: string; + data: unknown; + promptId?: string; + originatorClientId?: string; + }, + ): BackendEvent[] { + const data = isRecord(envelope.data) ? envelope.data : {}; + switch (envelope.type) { + case 'pending_prompt_started': { + state.busy = true; + state.turnBuffer = ''; + const jobRef = envelope.promptId ?? state.activeJobRef; + if (jobRef !== undefined) state.activeJobRef = jobRef; + return [ + { type: 'turn_started', ...(jobRef !== undefined ? { jobRef } : {}) }, + ]; + } + case 'session_update': { + const update = isRecord(data['update']) ? data['update'] : undefined; + if (!update) return []; + const kind = update['sessionUpdate']; + if (kind === 'agent_message_chunk') { + const content = isRecord(update['content']) + ? update['content'] + : undefined; + const text = content?.['text']; + if (typeof text === 'string') { + state.turnBuffer = `${state.turnBuffer}${text}`; + if (state.turnBuffer.length > MAX_DETAIL_CHARS) { + state.turnBuffer = tailSlice(state.turnBuffer, MAX_DETAIL_CHARS); + } + } + return []; + } + if (kind === 'tool_call') { + const title = update['title']; + const summary = + typeof title === 'string' ? sanitizeTitleLine(title) : ''; + return [ + { + type: 'progress', + ...(envelope.promptId !== undefined + ? { jobRef: envelope.promptId } + : {}), + summary: summary || 'running a tool', + }, + ]; + } + return []; + } + case 'turn_complete': { + state.busy = false; + const jobRef = envelope.promptId ?? state.activeJobRef; + state.activeJobRef = undefined; + const detail = state.turnBuffer.trim(); + state.turnBuffer = ''; + return [ + { + type: 'turn_complete', + ...(jobRef !== undefined ? { jobRef } : {}), + summary: clampTail(detail, MAX_SUMMARY_CHARS), + ...(detail ? { detail } : {}), + }, + ]; + } + case 'turn_error': + case 'prompt_cancelled': { + state.busy = false; + const jobRef = envelope.promptId ?? state.activeJobRef; + state.activeJobRef = undefined; + state.turnBuffer = ''; + const message = data['message'] ?? data['error']; + return [ + { + type: 'turn_error', + ...(jobRef !== undefined ? { jobRef } : {}), + error: + envelope.type === 'prompt_cancelled' + ? 'cancelled' + : typeof message === 'string' + ? message + : 'the task failed', + }, + ]; + } + case 'permission_request': { + const requestId = data['requestId']; + if (typeof requestId !== 'string') return []; + const rawOptions = Array.isArray(data['options']) + ? data['options'] + : []; + const options: PermissionOption[] = rawOptions.flatMap((raw) => { + if (!isRecord(raw) || typeof raw['optionId'] !== 'string') return []; + // Wire shape (ACP PermissionOption): optionId + name + kind. + const name = + typeof raw['name'] === 'string' ? raw['name'] : undefined; + const wireKind = + typeof raw['kind'] === 'string' ? raw['kind'] : undefined; + const classified = classifyOption(raw['optionId'], name, wireKind); + return [ + { + optionId: raw['optionId'], + ...(name !== undefined ? { label: name } : {}), + kind: classified.kind, + ...(classified.escalation !== undefined + ? { escalation: classified.escalation } + : {}), + }, + ]; + }); + state.permissionOptions.set(requestId, options); + return [ + { + type: 'permission_request', + requestId, + title: describeToolCall(data['toolCall']), + options, + payload: data, + }, + ]; + } + case 'permission_resolved': + case 'permission_already_resolved': { + const requestId = data['requestId']; + if (typeof requestId !== 'string') return []; + state.permissionOptions.delete(requestId); + // The daemon's originator stamp is authoritative when present: + // another client (WebShell) can resolve the request while our own + // vote is still mid-round-trip, and the short-circuit below would + // otherwise attribute THEIR resolution to us (skipping the spoken + // ask retraction). ownVotes only covers anonymous resolutions — + // adopted sessions carry no issued clientId to compare against. + const votedByUs = state.ownVotes.delete(requestId); + return [ + { + type: 'permission_resolved', + requestId, + byUs: + envelope.originatorClientId !== undefined + ? state.clientId !== undefined && + envelope.originatorClientId === state.clientId + : votedByUs, + }, + ]; + } + case 'session_died': + case 'session_closed': { + state.closed = true; + state.busy = false; + return [{ type: 'session_closed' }]; + } + default: + return []; + } + } +} diff --git a/packages/qwen-live/src/adaptor/types.ts b/packages/qwen-live/src/adaptor/types.ts new file mode 100644 index 00000000000..9eaf2264408 --- /dev/null +++ b/packages/qwen-live/src/adaptor/types.ts @@ -0,0 +1,146 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * The seam between the live orchestrator and a coding-agent backend. + * + * An adaptor owns one backend (a qwen serve daemon, an ACP agent, ...) and + * exposes the small primitive set the orchestrator needs. Capability bits + * describe what the backend can actually do so the orchestrator can degrade + * per feature instead of branching on adaptor names. + */ + +/** Opaque backend session identity, scoped to the owning adaptor. */ +export interface BackendHandle { + /** Adaptor-scoped stable id (for qwen serve: the session id). */ + readonly id: string; + /** Identifies the owning adaptor (e.g. 'qwen-code'). */ + readonly adaptor: string; +} + +export interface BackendCapabilities { + /** Whether an in-flight turn can accept an appended instruction. */ + steering: 'native' | 'queued' | 'none'; + /** Whether handoff prompts may carry image attachments. */ + imageInput: boolean; + /** Whether permission requests surface as events and accept votes. */ + permissionForwarding: boolean; + /** Whether the backend can push speech-worthy messages mid-turn. */ + proactiveSpeak: boolean; + /** Whether the backend enumerates existing sessions. */ + sessionList: boolean; + eventDelivery: 'stream' | 'per-turn' | 'poll' | 'reply-only'; +} + +export interface SessionSummary { + handle: BackendHandle; + label?: string; + cwd?: string; + state: 'idle' | 'busy' | 'closed' | 'unknown'; +} + +/** Receipt returned by prompt(): the task was accepted, not completed. */ +export interface PromptReceipt { + status: 'accepted' | 'queued' | 'rejected'; + /** Correlates later turn events with this prompt (qwen serve: promptId). */ + jobRef?: string; + /** Human note for the realtime model to relay ("queue full", ...). */ + note?: string; + /** For steering: the instruction joined the currently running turn. */ + joinedActiveTurn?: boolean; +} + +export type ContentBlock = + | { type: 'text'; text: string } + | { type: 'image'; mimeType: string; data: Uint8Array; name?: string }; + +export type PermissionOptionKind = 'proceed' | 'reject' | 'other'; + +export interface PermissionOption { + optionId: string; + label?: string; + kind: PermissionOptionKind; + /** + * Grant breadth, from the wire's structured kind (`allow_once` vs + * `allow_always`, `reject_once` vs `reject_always`). A bare voice + * "allow" must take the narrowest ('once') option on offer instead of + * persisting an always-allow rule. Absent when the backend gave no + * signal. + */ + escalation?: 'once' | 'always'; +} + +export type BackendEvent = + | { type: 'turn_started'; jobRef?: string } + | { type: 'progress'; jobRef?: string; summary: string } + | { type: 'speak'; text: string } + | { + type: 'permission_request'; + requestId: string; + title: string; + options: readonly PermissionOption[]; + payload?: unknown; + } + | { + type: 'permission_resolved'; + requestId: string; + /** True when this daemon's own vote settled it (vs WebShell etc.). */ + byUs: boolean; + } + | { type: 'turn_complete'; jobRef?: string; summary: string; detail?: string } + | { type: 'turn_error'; jobRef?: string; error: string } + | { type: 'session_closed' }; + +export type PermissionDecision = 'allow' | 'deny' | 'cancel'; + +export interface BackendAdaptor { + readonly name: string; + + capabilities(): BackendCapabilities; + + /** + * Verify the backend is reachable and supports what this adaptor needs. + * Throws a descriptive error when it does not — fail fast at daemon start. + */ + preflight(): Promise; + + createSession(opts?: { + cwd?: string; + label?: string; + }): Promise; + + listSessions(): Promise; + + /** + * Submit one turn. Resolves as soon as the backend admits the prompt. + * When `steer` is set and the session is busy, the adaptor attempts a + * mid-turn injection first and reports the outcome in the receipt. + */ + prompt( + handle: BackendHandle, + blocks: readonly ContentBlock[], + opts?: { steer?: boolean }, + ): Promise; + + /** Long-lived normalized event stream for one session. */ + events( + handle: BackendHandle, + opts?: { signal?: AbortSignal }, + ): AsyncIterable; + + /** True when the session currently has an active turn. */ + isBusy(handle: BackendHandle): boolean; + + cancel(handle: BackendHandle): Promise; + + respondPermission( + handle: BackendHandle, + requestId: string, + decision: PermissionDecision, + ): Promise<'delivered' | 'already_resolved'>; + + close(): Promise; +} diff --git a/packages/qwen-live/src/config.test.ts b/packages/qwen-live/src/config.test.ts new file mode 100644 index 00000000000..d8672d32933 --- /dev/null +++ b/packages/qwen-live/src/config.test.ts @@ -0,0 +1,179 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { homedir, tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, describe, expect, it } from 'vitest'; +import { loadConfig } from './config.js'; + +const temporaryDirectories: string[] = []; + +async function temporaryDataDir(): Promise { + const directory = await mkdtemp(join(tmpdir(), 'qwen-live-config-')); + temporaryDirectories.push(directory); + return directory; +} + +async function dataDirWithConfig(config: unknown): Promise { + const dataDir = await temporaryDataDir(); + await writeFile(join(dataDir, 'config.json'), JSON.stringify(config)); + return dataDir; +} + +function thrownMessage(callback: () => unknown): string { + try { + callback(); + } catch (error) { + return error instanceof Error ? error.message : String(error); + } + throw new Error('expected the callback to throw'); +} + +afterEach(async () => { + await Promise.all( + temporaryDirectories + .splice(0) + .map((directory) => rm(directory, { recursive: true, force: true })), + ); +}); + +describe('loadConfig', () => { + it('applies env over file over built-in defaults', async () => { + const dataDir = await dataDirWithConfig({ + realtimeApiKey: 'file-key', + realtimeModel: 'file-model', + voice: 'FileVoice', + port: 4171, + }); + + const envWins = loadConfig({ + QWEN_LIVE_DATA_DIR: dataDir, + DASHSCOPE_API_KEY: 'env-key', + QWEN_LIVE_REALTIME_MODEL: 'env-model', + QWEN_LIVE_VOICE: 'EnvVoice', + QWEN_LIVE_PORT: '4172', + }); + expect(envWins.realtime.apiKey).toBe('env-key'); + expect(envWins.realtime.model).toBe('env-model'); + expect(envWins.realtime.voice).toBe('EnvVoice'); + expect(envWins.port).toBe(4172); + + const fileWins = loadConfig({ QWEN_LIVE_DATA_DIR: dataDir }); + expect(fileWins.realtime.apiKey).toBe('file-key'); + expect(fileWins.realtime.model).toBe('file-model'); + expect(fileWins.realtime.voice).toBe('FileVoice'); + expect(fileWins.port).toBe(4171); + + const defaults = loadConfig({ + QWEN_LIVE_DATA_DIR: await temporaryDataDir(), + DASHSCOPE_API_KEY: 'env-key', + }); + expect(defaults.realtime.model).toBe('qwen3.5-omni-plus-realtime'); + expect(defaults.realtime.endpoint).toBe('https://dashscope.aliyuncs.com'); + expect(defaults.serve.baseUrl).toBe('http://127.0.0.1:4170'); + expect(defaults.port).toBe(0); + }); + + it('fails fast when no realtime API key is configured anywhere', async () => { + const dataDir = await temporaryDataDir(); + const message = thrownMessage(() => + loadConfig({ QWEN_LIVE_DATA_DIR: dataDir }), + ); + expect(message).toContain('DASHSCOPE_API_KEY'); + expect(message).toContain(join(dataDir, 'config.json')); + }); + + it('surfaces non-ENOENT config read errors naming the path', async () => { + // A config.json that is a DIRECTORY yields EISDIR on read — the + // portable stand-in for an unreadable file (chmod-based denial is + // unreliable when the suite runs as root). This must NOT be swallowed + // as "no config file". + const dataDir = await temporaryDataDir(); + await mkdir(join(dataDir, 'config.json')); + + const message = thrownMessage(() => + loadConfig({ QWEN_LIVE_DATA_DIR: dataDir, DASHSCOPE_API_KEY: 'key' }), + ); + expect(message).toContain( + `Could not read config file ${join(dataDir, 'config.json')}`, + ); + }); + + it('parses a config file saved with a UTF-8 BOM', async () => { + const dataDir = await temporaryDataDir(); + await writeFile( + join(dataDir, 'config.json'), + '\uFEFF' + JSON.stringify({ realtimeApiKey: 'bom-key', port: 4173 }), + ); + + const config = loadConfig({ QWEN_LIVE_DATA_DIR: dataDir }); + expect(config.realtime.apiKey).toBe('bom-key'); + expect(config.port).toBe(4173); + }); + + it('rejects wrong-typed file port values instead of defaulting to 0', async () => { + for (const port of [true, [4171], { value: 4171 }]) { + const dataDir = await dataDirWithConfig({ realtimeApiKey: 'k', port }); + const message = thrownMessage(() => + loadConfig({ QWEN_LIVE_DATA_DIR: dataDir }), + ); + expect(message).toContain(`Invalid "port" in`); + expect(message).toContain(JSON.stringify(port)); + } + }); + + it('names config.json, not QWEN_LIVE_PORT, for a file-sourced bad port', async () => { + const dataDir = await dataDirWithConfig({ + realtimeApiKey: 'k', + port: 'abc', + }); + const message = thrownMessage(() => + loadConfig({ QWEN_LIVE_DATA_DIR: dataDir }), + ); + expect(message).toContain( + `Invalid "port" in ${join(dataDir, 'config.json')}: "abc"`, + ); + expect(message).not.toContain('QWEN_LIVE_PORT'); + }); + + it('names QWEN_LIVE_PORT for an env-sourced bad port', async () => { + const dataDir = await dataDirWithConfig({ realtimeApiKey: 'k' }); + expect(() => + loadConfig({ QWEN_LIVE_DATA_DIR: dataDir, QWEN_LIVE_PORT: '70000' }), + ).toThrow('Invalid QWEN_LIVE_PORT: 70000'); + }); + + it('expands a leading ~ in dataDir, defaultCwd, and discoveryDir', async () => { + const dataDir = await dataDirWithConfig({ + realtimeApiKey: 'k', + discoveryDir: '~/qwen-live-test-discovery', + }); + + const config = loadConfig({ + QWEN_LIVE_DATA_DIR: dataDir, + QWEN_LIVE_CWD: '~/qwen-live-test-cwd', + }); + expect(config.defaultCwd).toBe(join(homedir(), 'qwen-live-test-cwd')); + expect(config.discoveryDir).toBe( + join(homedir(), 'qwen-live-test-discovery'), + ); + + // dataDir itself expands too; the missing-key error names the real path. + const message = thrownMessage(() => + loadConfig({ QWEN_LIVE_DATA_DIR: '~/qwen-live-test-nonexistent' }), + ); + expect(message).toContain( + join(homedir(), 'qwen-live-test-nonexistent', 'config.json'), + ); + }); + + it('defaults discoveryDir to the stable ~/.qwen base', async () => { + const dataDir = await dataDirWithConfig({ realtimeApiKey: 'k' }); + const config = loadConfig({ QWEN_LIVE_DATA_DIR: dataDir }); + expect(config.discoveryDir).toBe(join(homedir(), '.qwen')); + }); +}); diff --git a/packages/qwen-live/src/config.ts b/packages/qwen-live/src/config.ts new file mode 100644 index 00000000000..30f2adf3ea7 --- /dev/null +++ b/packages/qwen-live/src/config.ts @@ -0,0 +1,183 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Configuration: environment first, `~/.qwen-live/config.json` as fallback, + * built-in defaults last. Hand-rolled validation — the surface is small and + * a schema library would be the package's only heavy dependency. + */ + +import { readFileSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { join } from 'node:path'; +import { getStableLiveDiscoveryBaseDir } from './host/discovery.js'; + +export interface LiveConfig { + realtime: { + endpoint: string; + apiKey: string; + model: string; + voice?: string; + }; + serve: { + baseUrl: string; + token?: string; + }; + /** Default working directory for handoff-created sessions. */ + defaultCwd?: string; + /** Data root: session logs live in `/sessions`. */ + dataDir: string; + /** Where the Host discovery file lives (`~/.qwen` for the shipped Host). */ + discoveryDir: string; + /** Global shortcut advertised to the Host. */ + shortcut?: string; + /** Fixed listen port; 0 (default) lets the kernel pick. */ + port: number; +} + +const DEFAULT_REALTIME_ENDPOINT = 'https://dashscope.aliyuncs.com'; +const DEFAULT_REALTIME_MODEL = 'qwen3.5-omni-plus-realtime'; +const DEFAULT_SERVE_URL = 'http://127.0.0.1:4170'; + +function readConfigFile(path: string): Record { + let raw: string; + try { + raw = readFileSync(path, 'utf8'); + } catch (error) { + // Only a genuinely missing file means "no config". Anything else + // (EACCES, EISDIR, EIO) must surface, or the later missing-key error + // would point the user at a file that already contains the key. + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return {}; + throw new Error( + `Could not read config file ${path}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } + // Editors commonly save JSON with a UTF-8 BOM; JSON.parse rejects it. + if (raw.charCodeAt(0) === 0xfeff) raw = raw.slice(1); + try { + const parsed = JSON.parse(raw) as unknown; + if (typeof parsed === 'object' && parsed !== null && !Array.isArray(parsed)) + return parsed as Record; + throw new Error('not an object'); + } catch (error) { + throw new Error( + `Invalid config file ${path}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } +} + +function str(value: unknown): string | undefined { + return typeof value === 'string' && value.trim() ? value.trim() : undefined; +} + +/** Shell-style expansion of a leading `~` to the user's home directory. */ +function expandTilde(value: string): string { + if (value === '~') return homedir(); + if (value.startsWith('~/') || value.startsWith('~\\')) { + return join(homedir(), value.slice(2)); + } + return value; +} + +function pathStr(value: unknown): string | undefined { + const trimmed = str(value); + return trimmed === undefined ? undefined : expandTilde(trimmed); +} + +function resolvePort( + env: Record, + file: Record, + configPath: string, +): number { + const envPort = str(env['QWEN_LIVE_PORT']); + if (envPort !== undefined) { + const port = Number(envPort); + if (!Number.isInteger(port) || port < 0 || port > 65_535) { + throw new Error(`Invalid QWEN_LIVE_PORT: ${envPort}`); + } + return port; + } + const filePort = file['port']; + if (filePort === undefined) return 0; + // Accept the natural JSON spelling ("port": 4171) as well as a string. + // Any other type (true, [4171], {}) is a config mistake and must not + // silently boot on a kernel-picked ephemeral port. + const portRaw = + typeof filePort === 'number' + ? String(filePort) + : typeof filePort === 'string' + ? filePort.trim() + : undefined; + const port = portRaw ? Number(portRaw) : NaN; + if (!Number.isInteger(port) || port < 0 || port > 65_535) { + throw new Error( + `Invalid "port" in ${configPath}: ${JSON.stringify(filePort)}`, + ); + } + return port; +} + +export function loadConfig( + env: Record = process.env, +): LiveConfig { + const dataDir = + pathStr(env['QWEN_LIVE_DATA_DIR']) ?? join(homedir(), '.qwen-live'); + const configPath = join(dataDir, 'config.json'); + const file = readConfigFile(configPath); + + const apiKey = + str(env['DASHSCOPE_API_KEY']) ?? + str(env['QWEN_LIVE_REALTIME_API_KEY']) ?? + str(file['realtimeApiKey']); + if (!apiKey) { + throw new Error( + 'A DashScope realtime API key is required: set DASHSCOPE_API_KEY ' + + `or put "realtimeApiKey" in ${configPath}.`, + ); + } + + const port = resolvePort(env, file, configPath); + + const voice = str(env['QWEN_LIVE_VOICE']) ?? str(file['voice']) ?? 'Tina'; + const defaultCwd = + pathStr(env['QWEN_LIVE_CWD']) ?? pathStr(file['defaultCwd']); + const serveToken = str(env['QWEN_SERVER_TOKEN']) ?? str(file['serveToken']); + const shortcut = str(env['QWEN_LIVE_SHORTCUT']) ?? str(file['shortcut']); + + return { + realtime: { + endpoint: + str(env['QWEN_LIVE_REALTIME_ENDPOINT']) ?? + str(file['realtimeEndpoint']) ?? + DEFAULT_REALTIME_ENDPOINT, + apiKey, + model: + str(env['QWEN_LIVE_REALTIME_MODEL']) ?? + str(file['realtimeModel']) ?? + DEFAULT_REALTIME_MODEL, + ...(voice ? { voice } : {}), + }, + serve: { + baseUrl: + str(env['QWEN_LIVE_SERVE_URL']) ?? + str(file['serveUrl']) ?? + DEFAULT_SERVE_URL, + ...(serveToken ? { token: serveToken } : {}), + }, + ...(defaultCwd ? { defaultCwd } : {}), + dataDir, + discoveryDir: + pathStr(env['QWEN_LIVE_DISCOVERY_DIR']) ?? + pathStr(file['discoveryDir']) ?? + getStableLiveDiscoveryBaseDir(), + ...(shortcut ? { shortcut } : {}), + port, + }; +} diff --git a/packages/qwen-live/src/daemon.test.ts b/packages/qwen-live/src/daemon.test.ts new file mode 100644 index 00000000000..3ba20ca853c --- /dev/null +++ b/packages/qwen-live/src/daemon.test.ts @@ -0,0 +1,357 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { randomBytes } from 'node:crypto'; +import { request } from 'node:http'; +import type { Socket } from 'node:net'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import WebSocket from 'ws'; +import { afterEach, describe, expect, it } from 'vitest'; +import type { QwenCodeAdaptor } from './adaptor/qwen-code-adaptor.js'; +import type { LiveHostInstaller } from './host/live-host-installer.js'; +import type { LiveConfig } from './config.js'; +import { LiveDaemon } from './daemon.js'; +import { + getLiveDiscoveryPath, + type LiveDiscoveryRecord, +} from './host/discovery.js'; +import { LIVE_HOST_PROTOCOL_VERSION } from './host/types.js'; +import { LiveLogger } from './logger.js'; + +const temporaryDirectories: string[] = []; +const daemons: LiveDaemon[] = []; + +async function temporaryDirectory(): Promise { + const directory = await mkdtemp(join(tmpdir(), 'qwen-live-daemon-')); + temporaryDirectories.push(directory); + return directory; +} + +function fakeAdaptor(): QwenCodeAdaptor { + return { preflight: async () => undefined } as unknown as QwenCodeAdaptor; +} + +function fakeInstaller(): LiveHostInstaller { + return { + refresh: async () => ({ state: 'installed', version: '0.1.0' }), + ensureInstalled: async () => ({ state: 'installed', version: '0.1.0' }), + launch: async () => ({ state: 'installed', version: '0.1.0' }), + getStatus: () => ({ state: 'installed', version: '0.1.0' }), + } as unknown as LiveHostInstaller; +} + +async function testConfig(): Promise { + const base = await temporaryDirectory(); + return { + realtime: { + endpoint: 'https://dashscope.example.invalid', + apiKey: 'test-api-key', + model: 'test-model', + }, + serve: { baseUrl: 'http://127.0.0.1:1' }, + dataDir: join(base, 'data'), + discoveryDir: join(base, 'discovery'), + port: 0, + }; +} + +function startedDaemon( + config: LiveConfig, + deps: { installer?: LiveHostInstaller } = {}, +): LiveDaemon { + const daemon = new LiveDaemon(config, { + adaptor: fakeAdaptor(), + ...(deps.installer ? { installer: deps.installer } : {}), + // 'error' level keeps expected warnings (discovery cleanup) off stderr. + logger: new LiveLogger('error'), + }); + daemons.push(daemon); + return daemon; +} + +async function readDiscoveryRecord( + discoveryDir: string, +): Promise { + const raw = await readFile(getLiveDiscoveryPath(discoveryDir), 'utf8'); + return JSON.parse(raw) as LiveDiscoveryRecord; +} + +async function plantDiscoveryRecord( + discoveryDir: string, + record: LiveDiscoveryRecord, +): Promise { + const discoveryPath = getLiveDiscoveryPath(discoveryDir); + await mkdir(dirname(discoveryPath), { recursive: true, mode: 0o700 }); + await writeFile(discoveryPath, `${JSON.stringify(record)}\n`, { + mode: 0o600, + }); +} + +function delay(milliseconds: number): Promise { + return new Promise((resolve) => { + setTimeout(resolve, milliseconds).unref(); + }); +} + +function connectHost(url: string, headers: Record): WebSocket { + const target = url.replace(/^http/, 'ws'); + return new WebSocket(`${target}/live/host`, { headers }); +} + +function hostHeaders(record: LiveDiscoveryRecord): Record { + return { + authorization: `Bearer ${record.token}`, + 'x-qwen-live-nonce': record.instanceNonce, + }; +} + +function waitForOpen(socket: WebSocket): Promise { + return new Promise((resolve, reject) => { + socket.once('open', resolve); + socket.once('error', reject); + }); +} + +/** Resolves with the HTTP status of a refused upgrade, rejects on success. */ +function waitForRefusal(socket: WebSocket): Promise { + return new Promise((resolve, reject) => { + socket.once('unexpected-response', (_req, res) => { + resolve(res.statusCode ?? 'destroyed'); + socket.terminate(); + }); + socket.once('error', () => { + resolve('destroyed'); + }); + socket.once('open', () => { + reject(new Error('the upgrade was accepted')); + }); + }); +} + +afterEach(async () => { + await Promise.all(daemons.splice(0).map((daemon) => daemon.stop())); + await Promise.all( + temporaryDirectories + .splice(0) + .map((directory) => rm(directory, { recursive: true, force: true })), + ); +}); + +describe('LiveDaemon', () => { + it('stop() without start() resolves quickly', async () => { + const daemon = startedDaemon(await testConfig()); + const outcome = await Promise.race([ + daemon.stop().then(() => 'stopped' as const), + delay(2_000).then(() => 'timeout' as const), + ]); + expect(outcome).toBe('stopped'); + }); + + it('publishes a discovery record and accepts a Host presenting it', async () => { + const config = await testConfig(); + const daemon = startedDaemon(config); + const { url, port } = await daemon.start(); + expect(url).toBe(`http://127.0.0.1:${port}`); + + const record = await readDiscoveryRecord(config.discoveryDir); + expect(record.pid).toBe(process.pid); + expect(record.url).toBe(url); + expect(record.protocolVersion).toBe(LIVE_HOST_PROTOCOL_VERSION); + + const socket = connectHost(url, hostHeaders(record)); + await waitForOpen(socket); + expect(socket.readyState).toBe(WebSocket.OPEN); + socket.terminate(); + }); + + it('refuses an upgrade that carries an Origin header (CSRF wall)', async () => { + const config = await testConfig(); + const daemon = startedDaemon(config); + const { url } = await daemon.start(); + const record = await readDiscoveryRecord(config.discoveryDir); + + // Even a correct token is refused when Origin marks a browser context. + const socket = connectHost(url, { + ...hostHeaders(record), + origin: 'https://evil.example.com', + }); + await expect(waitForRefusal(socket)).resolves.toBe(401); + }); + + it('refuses a wrong bearer token with 401', async () => { + const config = await testConfig(); + const daemon = startedDaemon(config); + const { url } = await daemon.start(); + const record = await readDiscoveryRecord(config.discoveryDir); + + const socket = connectHost(url, { + ...hostHeaders(record), + authorization: 'Bearer not-the-token', + }); + await expect(waitForRefusal(socket)).resolves.toBe(401); + }); + + it('stop() resolves within a deadline while a Host keeps reconnecting', async () => { + const config = await testConfig(); + const daemon = startedDaemon(config); + const { port } = await daemon.start(); + const record = await readDiscoveryRecord(config.discoveryDir); + + // A raw upgraded socket that never answers the server's close frame — + // the worst-case peer graceful close would wait on. Resolves with the + // socket once the 101 handshake completed, or undefined when refused. + const rawHostSocket = (): Promise => + new Promise((resolve) => { + const req = request({ + host: '127.0.0.1', + port, + path: '/live/host', + headers: { + ...hostHeaders(record), + Connection: 'Upgrade', + Upgrade: 'websocket', + 'Sec-WebSocket-Version': '13', + 'Sec-WebSocket-Key': randomBytes(16).toString('base64'), + }, + }); + req.on('upgrade', (_res, socket) => { + socket.on('error', () => {}); + resolve(socket); + }); + req.on('response', () => { + resolve(undefined); + }); + req.on('error', () => { + resolve(undefined); + }); + req.end(); + }); + + const sockets: Socket[] = []; + let redialing = true; + const redial = (socket: Socket) => { + socket.on('close', () => { + if (!redialing) return; + setTimeout(() => { + if (!redialing) return; + void rawHostSocket().then((next) => { + if (!next) return; + sockets.push(next); + redial(next); + }); + }, 5).unref(); + }); + }; + + const first = await rawHostSocket(); + expect(first).toBeDefined(); + sockets.push(first!); + redial(first!); + + try { + const outcome = await Promise.race([ + daemon.stop().then(() => 'stopped' as const), + delay(3_000).then(() => 'timeout' as const), + ]); + expect(outcome).toBe('stopped'); + } finally { + redialing = false; + for (const socket of sockets) socket.destroy(); + } + }); + + it('reclaims a stale dead-owner discovery record and republishes', async () => { + const config = await testConfig(); + await plantDiscoveryRecord(config.discoveryDir, { + url: 'http://127.0.0.1:3210', + token: 'stale-token', + protocolVersion: LIVE_HOST_PROTOCOL_VERSION, + pid: 999_999, + instanceNonce: 'daemon_instance_nonce_stale_01', + }); + + const daemon = startedDaemon(config); + const { url } = await daemon.start(); + + const record = await readDiscoveryRecord(config.discoveryDir); + expect(record.pid).toBe(process.pid); + expect(record.url).toBe(url); + expect(record.instanceNonce).not.toBe('daemon_instance_nonce_stale_01'); + expect(record.token).not.toBe('stale-token'); + }); + + it('fails fast on a live discovery owner instead of stealing the file', async () => { + const config = await testConfig(); + const planted: LiveDiscoveryRecord = { + url: 'http://127.0.0.1:3210', + token: 'live-owner-token', + protocolVersion: LIVE_HOST_PROTOCOL_VERSION, + // This test process is alive, so the record reads as a live owner. + pid: process.pid, + instanceNonce: 'daemon_instance_nonce_live_01', + }; + await plantDiscoveryRecord(config.discoveryDir, planted); + + const daemon = startedDaemon(config); + await expect(daemon.start()).rejects.toThrow( + /Another Live daemon \(pid \d+\) already owns/, + ); + // The live owner's record is untouched. + await expect(readDiscoveryRecord(config.discoveryDir)).resolves.toEqual( + planted, + ); + }); + + it('serves the installer status behind the bearer wall', async () => { + const config = await testConfig(); + const daemon = startedDaemon(config, { installer: fakeInstaller() }); + const { port } = await daemon.start(); + const record = await readDiscoveryRecord(config.discoveryDir); + + const get = (path: string, headers: Record) => + new Promise<{ status: number; body: string }>((resolve, reject) => { + const req = request( + { host: '127.0.0.1', port, path, headers }, + (res) => { + let body = ''; + res.on('data', (chunk: Buffer) => { + body += chunk.toString(); + }); + res.on('end', () => resolve({ status: res.statusCode ?? 0, body })); + }, + ); + req.on('error', reject); + req.end(); + }); + + // Authorized: status JSON comes back. + const ok = await get('/live/setup', hostHeaders(record)); + expect(ok.status).toBe(200); + expect(JSON.parse(ok.body)).toMatchObject({ + state: 'installed', + version: '0.1.0', + }); + + // Wrong token: 401, no body leakage. + const denied = await get('/live/setup', { + authorization: 'Bearer wrong-token', + }); + expect(denied.status).toBe(401); + + // A browser context (Origin) is refused even with the right token. + const browser = await get('/live/setup', { + ...hostHeaders(record), + origin: 'https://evil.example', + }); + expect(browser.status).toBe(401); + + // Unknown paths stay 404. + const missing = await get('/nope', hostHeaders(record)); + expect(missing.status).toBe(404); + }); +}); diff --git a/packages/qwen-live/src/daemon.ts b/packages/qwen-live/src/daemon.ts new file mode 100644 index 00000000000..81104c23923 --- /dev/null +++ b/packages/qwen-live/src/daemon.ts @@ -0,0 +1,336 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * LiveDaemon wires everything together: the Host WebSocket endpoint + * (protocol v6, same wire contract as the shipped Qwen Live Host), the + * discovery file that Host binaries poll, the realtime orchestrator, and + * the qwen serve adaptor. + * + * Discovery mutual exclusion: writing `~/.qwen/live/daemon.json` fails fast + * when a live owner (typically qwen serve's built-in Live integration) + * already holds it — the two must not fight over the Host. + */ + +import { randomUUID, timingSafeEqual } from 'node:crypto'; +import { createServer, type IncomingMessage, type Server } from 'node:http'; +import { join } from 'node:path'; +import { WebSocketServer, type WebSocket } from 'ws'; +import { QwenCodeAdaptor } from './adaptor/qwen-code-adaptor.js'; +import { LiveHostInstaller } from './host/live-host-installer.js'; +import type { LiveConfig } from './config.js'; +import { + handoffLiveDiscoveryOwner, + LiveDiscoveryOwnerActiveError, + removeLiveDiscoveryFile, + writeLiveDiscoveryFile, +} from './host/discovery.js'; +import { LiveHostCoordinator } from './host/live-host-coordinator.js'; +import { LIVE_HOST_PROTOCOL_VERSION } from './host/types.js'; +import { SessionLog } from './log/session-log.js'; +import { LiveLogger } from './logger.js'; +import { LiveSession } from './orchestrator/live-session.js'; + +const HOST_WS_PATH = '/live/host'; + +export interface LiveDaemonDeps { + adaptor?: QwenCodeAdaptor; + /** Installer (tests inject a fake); production installs the real Host. */ + installer?: LiveHostInstaller; + logger?: LiveLogger; +} + +export class LiveDaemon { + private readonly logger: LiveLogger; + private readonly adaptor: QwenCodeAdaptor; + private readonly installer: LiveHostInstaller; + private readonly token = randomUUID(); + private readonly instanceNonce = randomUUID(); + private server: Server | undefined; + private wss: WebSocketServer | undefined; + private coordinator: LiveHostCoordinator | undefined; + private session: LiveSession | undefined; + private log: SessionLog | undefined; + private discoveryPublished = false; + private stopping = false; + + constructor( + private readonly config: LiveConfig, + deps: LiveDaemonDeps = {}, + ) { + this.logger = deps.logger ?? new LiveLogger(); + this.installer = deps.installer ?? new LiveHostInstaller(); + this.adaptor = + deps.adaptor ?? + new QwenCodeAdaptor({ + baseUrl: config.serve.baseUrl, + ...(config.serve.token ? { token: config.serve.token } : {}), + ...(config.defaultCwd ? { defaultCwd: config.defaultCwd } : {}), + clientId: `qwen-live-${this.instanceNonce.slice(0, 8)}`, + }); + } + + async start(): Promise<{ port: number; url: string }> { + // Fail fast when qwen serve is missing or too old — before we take the + // Host discovery file from anyone. + await this.adaptor.preflight(); + + const coordinator = new LiveHostCoordinator({ + daemonInstanceNonce: this.instanceNonce, + ...(this.config.shortcut ? { shortcut: this.config.shortcut } : {}), + getProviderReadiness: () => + this.config.realtime.apiKey + ? { state: 'ready' } + : { + state: 'unavailable', + blocker: 'provider_config', + message: 'DashScope realtime API key is not configured.', + }, + }); + this.coordinator = coordinator; + // The ported coordinator fails closed until the Appshot delivery channel + // is verified (in qwen serve that channel is a separate reverse-RPC hop + // booted lazily). Here the channel is the in-process + // captureScreenContext call, verified by construction. + coordinator.setAppshotReadiness({ state: 'ready' }); + + const log = new SessionLog({ + directory: join(this.config.dataDir, 'sessions'), + liveSessionId: `live-${Date.now()}-${this.instanceNonce.slice(0, 8)}`, + }); + this.log = log; + + const session = new LiveSession({ + host: coordinator, + adaptor: this.adaptor, + realtime: { + endpoint: this.config.realtime.endpoint, + apiKey: this.config.realtime.apiKey, + model: this.config.realtime.model, + ...(this.config.realtime.voice + ? { voice: this.config.realtime.voice } + : {}), + }, + log, + }); + this.session = session; + + coordinator.setHandlers({ + onStart: (call) => session.start(call), + onStop: (call) => session.stop(call), + onInputAudio: (call) => session.pushAudio(call), + }); + + const port = await this.listen(); + const url = `http://127.0.0.1:${port}`; + + await this.publishDiscovery(url); + + // The single machine-readable stdout line; harnesses parse the port + // from it (same pattern as `qwen serve`). + process.stdout.write(`qwen-live listening on ${url}\n`); + this.logger.info( + `host endpoint ready at ${url}${HOST_WS_PATH} (protocol v${LIVE_HOST_PROTOCOL_VERSION})`, + ); + return { port, url }; + } + + async stop(): Promise { + if (this.stopping) return; + this.stopping = true; + this.session?.dispose(); + this.coordinator?.dispose(); + if (this.discoveryPublished) { + try { + await removeLiveDiscoveryFile(this.config.discoveryDir, { + pid: process.pid, + instanceNonce: this.instanceNonce, + }); + } catch (error) { + this.logger.warn( + `could not remove the discovery file: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } + } + await this.log?.close(); + // Graceful close waits on the peer; shutdown must not. Any client still + // attached (or attached between dispose() and here) is torn down hard. + if (this.wss) { + for (const client of this.wss.clients) { + client.terminate(); + } + } + await new Promise((resolve) => { + if (!this.wss) { + resolve(); + return; + } + this.wss.close(() => { + resolve(); + }); + }); + await new Promise((resolve) => { + if (!this.server) { + resolve(); + return; + } + this.server.close(() => { + resolve(); + }); + }); + } + + // -- internals ------------------------------------------------------------ + + /** + * Minimal HTTP surface for Host bootstrap: the installer endpoints. The + * same Bearer wall as the WS upgrade guards them — a browser context + * (Origin) is always refused. + */ + private handleRequest( + req: IncomingMessage, + res: import('node:http').ServerResponse, + ): void { + const url = (req.url ?? '').split('?', 1)[0]; + const route = `${req.method} ${url}`; + if ( + (route === 'GET /live/setup' || + route === 'POST /live/setup/install' || + route === 'POST /live/setup/launch') && + this.authorize(req) + ) { + void this.serveSetup(route, res); + return; + } + if (route === 'GET /healthz') { + res.statusCode = 200; + res.end('ok'); + return; + } + res.statusCode = route.startsWith('GET /live/setup') ? 401 : 404; + res.end(); + } + + private async serveSetup( + route: string, + res: import('node:http').ServerResponse, + ): Promise { + const status = + route === 'GET /live/setup' + ? await this.installer.refresh() + : route === 'POST /live/setup/install' + ? // force=true is the recovery path: a corrupted-but-present + // installation (bad Info.plist, revoked notarization) would + // otherwise permanently fail inspection while the status still + // says retryable. Force re-downloads and re-verifies. + await this.installer.ensureInstalled(true) + : await this.installer.launch(); + res.statusCode = 200; + res.setHeader('content-type', 'application/json'); + res.end(JSON.stringify(status)); + } + + private authorize(req: IncomingMessage): boolean { + // CSRF wall: a browser context always sends Origin; the Host never does. + if (req.headers['origin'] !== undefined) return false; + const auth = req.headers['authorization']; + if (typeof auth !== 'string' || !auth.startsWith('Bearer ')) return false; + const presented = Buffer.from(auth.slice('Bearer '.length)); + const expected = Buffer.from(this.token); + return ( + presented.length === expected.length && + timingSafeEqual(presented, expected) + ); + } + + private listen(): Promise { + const server = createServer((req, res) => { + this.handleRequest(req, res); + }); + this.server = server; + const wss = new WebSocketServer({ noServer: true }); + this.wss = wss; + + server.on('upgrade', (req, socket, head) => { + // A Host redialing its cached discovery URL during shutdown must not + // re-register — stop() would then wait on the fresh lease forever. + if (this.stopping) { + socket.destroy(); + return; + } + const path = (req.url ?? '').split('?', 1)[0]; + if (path !== HOST_WS_PATH) { + socket.destroy(); + return; + } + if (!this.authorize(req)) { + socket.write('HTTP/1.1 401 Unauthorized\r\n\r\n'); + socket.destroy(); + return; + } + wss.handleUpgrade(req, socket, head, (ws: WebSocket) => { + const nonce = req.headers['x-qwen-live-nonce']; + this.coordinator?.attachHost( + ws, + typeof nonce === 'string' ? nonce : undefined, + ); + }); + }); + + return new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(this.config.port, '127.0.0.1', () => { + const address = server.address(); + if (address === null || typeof address === 'string') { + reject(new Error('could not determine the listening port')); + return; + } + resolve(address.port); + }); + }); + } + + private async publishDiscovery(url: string): Promise { + const record = { + url, + token: this.token, + protocolVersion: LIVE_HOST_PROTOCOL_VERSION, + pid: process.pid, + instanceNonce: this.instanceNonce, + }; + try { + await writeLiveDiscoveryFile(this.config.discoveryDir, record); + this.discoveryPublished = true; + return; + } catch (error) { + if (!(error instanceof LiveDiscoveryOwnerActiveError)) { + // A dead owner's record can be reclaimed through the handoff path. + // The handoff's commitOwner must not touch the discovery file: the + // handoff holds the directory lock (a nested write would deadlock) + // and its confirm step expects the stale record to still be there. + // Publish the new record after the reclaim completes — the same + // contract qwen serve's wiring follows. + await handoffLiveDiscoveryOwner( + this.config.discoveryDir, + { pid: process.pid, instanceNonce: this.instanceNonce }, + async () => undefined, + { waitForHandoffGrace: false }, + ); + await writeLiveDiscoveryFile(this.config.discoveryDir, record); + this.discoveryPublished = true; + return; + } + throw new Error( + `Another Live daemon (pid ${error.ownerPid}) already owns the Host ` + + 'discovery file — most likely qwen serve with its built-in Live ' + + 'Voice enabled. Stop it (or disable its Live integration) and ' + + 'start qwen-live again.', + ); + } + } +} diff --git a/packages/qwen-live/src/host/discovery.test.ts b/packages/qwen-live/src/host/discovery.test.ts new file mode 100644 index 00000000000..c3f8dedd72b --- /dev/null +++ b/packages/qwen-live/src/host/discovery.test.ts @@ -0,0 +1,346 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import * as fs from 'node:fs/promises'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { + getLiveDiscoveryPath, + handoffLiveDiscoveryOwner, + LiveDiscoveryOwnerActiveError, + LiveDiscoveryStateError, + removeLiveDiscoveryFile, + writeLiveDiscoveryFile, + type LiveDiscoveryRecord, +} from './discovery.js'; +import { LIVE_HOST_PROTOCOL_VERSION } from './types.js'; + +const temporaryDirectories: string[] = []; + +async function temporaryRuntime(): Promise { + const directory = await fs.mkdtemp(path.join(os.tmpdir(), 'qwen-live-')); + temporaryDirectories.push(directory); + return directory; +} + +function record(instanceNonce: string, pid = process.pid): LiveDiscoveryRecord { + return { + url: 'http://127.0.0.1:3210', + token: 'not-a-real-token', + protocolVersion: LIVE_HOST_PROTOCOL_VERSION, + pid, + instanceNonce, + }; +} + +afterEach(async () => { + await Promise.all( + temporaryDirectories + .splice(0) + .map((directory) => fs.rm(directory, { recursive: true, force: true })), + ); +}); + +describe('Live discovery file', () => { + it('publishes an atomic mode-0600 record below the runtime directory', async () => { + const runtime = await temporaryRuntime(); + const expected = record('daemon_instance_nonce_0001'); + + const writtenPath = await writeLiveDiscoveryFile(runtime, expected); + + expect(writtenPath).toBe(getLiveDiscoveryPath(runtime)); + expect(JSON.parse(await fs.readFile(writtenPath, 'utf8'))).toEqual( + expected, + ); + // Windows has no POSIX permission bits; the read side already skips + // the mode check there (see discovery.ts). + if (process.platform !== 'win32') { + expect((await fs.stat(writtenPath)).mode & 0o777).toBe(0o600); + } + expect( + (await fs.readdir(path.dirname(writtenPath))).filter((name) => + name.endsWith('.tmp'), + ), + ).toEqual([]); + }); + + it('safely creates a missing nested runtime directory tree', async () => { + const parent = await temporaryRuntime(); + const runtime = path.join(parent, 'nested', 'runtime', 'base'); + const expected = record('daemon_instance_nonce_nested_01'); + + await expect(writeLiveDiscoveryFile(runtime, expected)).resolves.toBe( + getLiveDiscoveryPath(runtime), + ); + await expect( + fs.readFile(getLiveDiscoveryPath(runtime), 'utf8'), + ).resolves.toContain(expected.instanceNonce); + if (process.platform !== 'win32') { + expect( + (await fs.stat(path.dirname(getLiveDiscoveryPath(runtime)))).mode & + 0o777, + ).toBe(0o700); + } + }); + + it('rejects a symlinked runtime base without publishing through it', async () => { + if (process.platform === 'win32') return; + const parent = await temporaryRuntime(); + const target = path.join(parent, 'target'); + const runtimeBaseDir = path.join(parent, 'runtime-link'); + await fs.mkdir(target, { mode: 0o700 }); + await fs.symlink(target, runtimeBaseDir); + + await expect( + writeLiveDiscoveryFile( + runtimeBaseDir, + record('daemon_instance_nonce_symlink_01'), + ), + ).rejects.toBeInstanceOf(LiveDiscoveryStateError); + await expect(fs.readdir(target)).resolves.toEqual([]); + }); + + it('rejects an unsafe discovery lock shape without replacing it', async () => { + const runtime = await temporaryRuntime(); + const discoveryDirectory = path.dirname(getLiveDiscoveryPath(runtime)); + const lockPath = path.join(discoveryDirectory, '.daemon.lock'); + await fs.mkdir(discoveryDirectory, { mode: 0o700 }); + await fs.writeFile(lockPath, 'unsafe', { mode: 0o600 }); + + await expect( + writeLiveDiscoveryFile( + runtime, + record('daemon_instance_nonce_unsafe_lock_01'), + ), + ).rejects.toBeInstanceOf(LiveDiscoveryStateError); + + await expect(fs.readFile(lockPath, 'utf8')).resolves.toBe('unsafe'); + await expect(fs.stat(getLiveDiscoveryPath(runtime))).rejects.toMatchObject({ + code: 'ENOENT', + }); + }); + + it('only removes a record owned by the matching daemon pid and nonce', async () => { + const runtime = await temporaryRuntime(); + const current = record('daemon_instance_nonce_0002'); + await writeLiveDiscoveryFile(runtime, current); + + await expect( + removeLiveDiscoveryFile(runtime, { + pid: current.pid, + instanceNonce: 'daemon_instance_nonce_old0', + }), + ).rejects.toBeInstanceOf(LiveDiscoveryStateError); + await expect( + removeLiveDiscoveryFile(runtime, { + pid: current.pid + 1, + instanceNonce: current.instanceNonce, + }), + ).rejects.toBeInstanceOf(LiveDiscoveryStateError); + await expect( + fs.readFile(getLiveDiscoveryPath(runtime), 'utf8'), + ).resolves.toContain(current.instanceNonce); + + expect(await removeLiveDiscoveryFile(runtime, current)).toBe(true); + await expect(fs.stat(getLiveDiscoveryPath(runtime))).rejects.toMatchObject({ + code: 'ENOENT', + }); + }); + + it('allows only one concurrent healthy owner', async () => { + const runtime = await temporaryRuntime(); + const first = record('daemon_instance_nonce_0003'); + const second = record('daemon_instance_nonce_0004'); + + const results = await Promise.allSettled([ + writeLiveDiscoveryFile(runtime, first), + writeLiveDiscoveryFile(runtime, second), + ]); + + expect( + results.filter((result) => result.status === 'fulfilled'), + ).toHaveLength(1); + const rejected = results.find((result) => result.status === 'rejected'); + expect(rejected).toMatchObject({ + status: 'rejected', + reason: expect.any(LiveDiscoveryOwnerActiveError), + }); + const published = JSON.parse( + await fs.readFile(getLiveDiscoveryPath(runtime), 'utf8'), + ) as LiveDiscoveryRecord; + expect([first.instanceNonce, second.instanceNonce]).toContain( + published.instanceNonce, + ); + }); + + it('does not publish over a stale foreign owner without a handoff', async () => { + const runtime = await temporaryRuntime(); + const previous = record('daemon_instance_nonce_0005', 999_999); + const replacement = record('daemon_instance_nonce_0006'); + await writeLiveDiscoveryFile(runtime, previous); + + await expect( + writeLiveDiscoveryFile(runtime, replacement, { + isProcessAlive: (pid) => { + expect(pid).toBe(previous.pid); + return false; + }, + }), + ).rejects.toBeInstanceOf(LiveDiscoveryStateError); + expect( + JSON.parse(await fs.readFile(getLiveDiscoveryPath(runtime), 'utf8')), + ).toEqual(previous); + }); + + it('does not publish over a stale legacy-protocol owner without a handoff', async () => { + const runtime = await temporaryRuntime(); + const discoveryPath = getLiveDiscoveryPath(runtime); + const previous = { + ...record('daemon_instance_nonce_legacy_01', 999_998), + protocolVersion: 2, + }; + const replacement = record('daemon_instance_nonce_current_01'); + await fs.mkdir(path.dirname(discoveryPath), { + recursive: true, + mode: 0o700, + }); + await fs.writeFile(discoveryPath, `${JSON.stringify(previous)}\n`, { + mode: 0o600, + }); + + await expect( + writeLiveDiscoveryFile(runtime, replacement, { + isProcessAlive: (pid) => { + expect(pid).toBe(previous.pid); + return false; + }, + }), + ).rejects.toBeInstanceOf(LiveDiscoveryStateError); + expect(JSON.parse(await fs.readFile(discoveryPath, 'utf8'))).toEqual( + previous, + ); + if (process.platform !== 'win32') { + expect((await fs.stat(discoveryPath)).mode & 0o777).toBe(0o600); + } + }); + + it.each([ + ['current', LIVE_HOST_PROTOCOL_VERSION, 999_996], + ['legacy', 2, 999_995], + ])( + 'reclaims a valid stale %s-protocol owner during handoff', + async (_label, protocolVersion, pid) => { + const runtime = await temporaryRuntime(); + const discoveryPath = getLiveDiscoveryPath(runtime); + const previous = { + ...record('daemon_instance_nonce_handoff_old', pid), + protocolVersion, + }; + const replacement = record('daemon_instance_nonce_handoff_new'); + await fs.mkdir(path.dirname(discoveryPath), { + recursive: true, + mode: 0o700, + }); + await fs.writeFile(discoveryPath, `${JSON.stringify(previous)}\n`, { + mode: 0o600, + }); + let committed = false; + const wait = vi.fn(async () => undefined); + + await expect( + handoffLiveDiscoveryOwner( + runtime, + replacement, + async () => { + committed = true; + }, + { + wait, + handoffGraceMs: 37, + isProcessAlive: (ownerPid) => { + expect(ownerPid).toBe(previous.pid); + return false; + }, + }, + ), + ).resolves.toEqual({ reclaimed: true }); + expect(committed).toBe(true); + expect(wait).toHaveBeenCalledOnce(); + expect(wait).toHaveBeenCalledWith(37); + await expect(fs.stat(discoveryPath)).rejects.toMatchObject({ + code: 'ENOENT', + }); + }, + ); + + it('does not replace a live owner only because its protocol is legacy', async () => { + const runtime = await temporaryRuntime(); + const discoveryPath = getLiveDiscoveryPath(runtime); + const previous = { + ...record('daemon_instance_nonce_legacy_02', 999_997), + protocolVersion: 2, + }; + await fs.mkdir(path.dirname(discoveryPath), { + recursive: true, + mode: 0o700, + }); + await fs.writeFile(discoveryPath, `${JSON.stringify(previous)}\n`, { + mode: 0o600, + }); + + await expect( + writeLiveDiscoveryFile( + runtime, + record('daemon_instance_nonce_current_02'), + { + isProcessAlive: (pid) => { + expect(pid).toBe(previous.pid); + return true; + }, + }, + ), + ).rejects.toBeInstanceOf(LiveDiscoveryOwnerActiveError); + expect(JSON.parse(await fs.readFile(discoveryPath, 'utf8'))).toEqual( + previous, + ); + }); + + it('does not overwrite an invalid existing record', async () => { + const runtime = await temporaryRuntime(); + const discoveryPath = getLiveDiscoveryPath(runtime); + await fs.mkdir(path.dirname(discoveryPath), { recursive: true }); + await fs.writeFile(discoveryPath, '{broken', { mode: 0o600 }); + + await expect( + writeLiveDiscoveryFile(runtime, record('daemon_instance_nonce_0007')), + ).rejects.toThrow('Existing Live discovery record is invalid.'); + await expect(fs.readFile(discoveryPath, 'utf8')).resolves.toBe('{broken'); + }); + + it('rejects non-loopback URLs and malformed nonces', async () => { + const runtime = await temporaryRuntime(); + await expect( + writeLiveDiscoveryFile(runtime, { + ...record('daemon_instance_nonce_0005'), + url: 'https://example.com', + }), + ).rejects.toThrow(/loopback/); + await expect( + writeLiveDiscoveryFile(runtime, record('short')), + ).rejects.toThrow(/invalid/); + }); + + it('accepts concrete IPv4 loopback addresses in 127/8', async () => { + const runtime = await temporaryRuntime(); + await expect( + writeLiveDiscoveryFile(runtime, { + ...record('daemon_instance_nonce_0006'), + url: 'http://127.12.34.56:3210', + }), + ).resolves.toBe(getLiveDiscoveryPath(runtime)); + }); +}); diff --git a/packages/qwen-live/src/host/discovery.ts b/packages/qwen-live/src/host/discovery.ts new file mode 100644 index 00000000000..4c4b5dcdb58 --- /dev/null +++ b/packages/qwen-live/src/host/discovery.ts @@ -0,0 +1,618 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { randomUUID } from 'node:crypto'; +import { constants as fsConstants } from 'node:fs'; +import * as fs from 'node:fs/promises'; +import { isIP } from 'node:net'; +import { homedir } from 'node:os'; +import * as path from 'node:path'; +import type { LockOptions } from 'proper-lockfile'; +import { LIVE_HOST_PROTOCOL_VERSION } from './types.js'; + +export const LIVE_DISCOVERY_RELATIVE_PATH = path.join('live', 'daemon.json'); +const MAX_DISCOVERY_BYTES = 16 * 1024; +const NONCE_PATTERN = /^[A-Za-z0-9_-]{16,256}$/; +const DEFAULT_HANDOFF_GRACE_MS = 1_000; +const LOCK_OPTIONS: LockOptions = { + realpath: false, + stale: 10_000, + update: 2_000, + retries: { + retries: 120, + minTimeout: 10, + maxTimeout: 100, + factor: 1.2, + randomize: true, + }, +}; + +export interface LiveDiscoveryRecord { + url: string; + token?: string; + protocolVersion: typeof LIVE_HOST_PROTOCOL_VERSION; + pid: number; + instanceNonce: string; +} + +export interface LiveDiscoveryOwner { + pid: number; + instanceNonce: string; +} + +interface ExistingLiveDiscoveryRecord { + url: string; + token?: string; + protocolVersion: number; + pid: number; + instanceNonce: string; +} + +interface LiveDiscoveryDirectory { + directory: string; + dev: number | bigint; + ino: number | bigint; +} + +export class LiveDiscoveryOwnerActiveError extends Error { + constructor(readonly ownerPid: number) { + super(`Live discovery is owned by active daemon pid ${ownerPid}.`); + this.name = 'LiveDiscoveryOwnerActiveError'; + } +} + +export class LiveDiscoveryStateError extends Error { + constructor(cause?: unknown) { + super('Existing Live discovery record is invalid.', { cause }); + this.name = 'LiveDiscoveryStateError'; + } +} + +export class LiveDiscoveryPublicationError extends Error { + readonly published = true; + + constructor(cause: unknown) { + super('Live discovery publication completed with cleanup errors.', { + cause, + }); + this.name = 'LiveDiscoveryPublicationError'; + } +} + +export function getStableLiveDiscoveryBaseDir( + homeDirectory = homedir(), +): string { + return path.join(homeDirectory, '.qwen'); +} + +export function getLiveDiscoveryPath(runtimeBaseDir: string): string { + return path.join(runtimeBaseDir, LIVE_DISCOVERY_RELATIVE_PATH); +} + +function assertSafeRecord(record: ExistingLiveDiscoveryRecord): void { + if ( + typeof record.url !== 'string' || + typeof record.instanceNonce !== 'string' || + (record.token !== undefined && typeof record.token !== 'string') + ) { + throw new Error('Live discovery record is invalid.'); + } + let url: URL; + try { + url = new URL(record.url); + } catch { + throw new Error('Live discovery URL is invalid.'); + } + const hostname = url.hostname.replace(/^\[|\]$/g, '').toLowerCase(); + const loopback = + hostname === 'localhost' || + hostname === '::1' || + (isIP(hostname) === 4 && hostname.startsWith('127.')); + if ( + (url.protocol !== 'http:' && url.protocol !== 'https:') || + !loopback || + url.username.length > 0 || + url.password.length > 0 + ) { + throw new Error('Live discovery URL must be a loopback HTTP URL.'); + } + if ( + !Number.isSafeInteger(record.protocolVersion) || + record.protocolVersion <= 0 || + !Number.isSafeInteger(record.pid) || + record.pid <= 0 || + !NONCE_PATTERN.test(record.instanceNonce) || + (record.token !== undefined && + (record.token.length === 0 || record.token.length > 4096)) + ) { + throw new Error('Live discovery record is invalid.'); + } +} + +function assertRecord(record: LiveDiscoveryRecord): void { + assertSafeRecord(record); + if (record.protocolVersion !== LIVE_HOST_PROTOCOL_VERSION) { + throw new Error('Live discovery record is invalid.'); + } +} + +function processIsAlive(pid: number): boolean { + // PID reuse is intentionally fail-closed. The locator has no independent + // authenticated nonce probe, so a live replacement process must not be + // mistaken for proof that the recorded daemon is stale. + try { + process.kill(pid, 0); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code !== 'ESRCH'; + } +} + +function delay(milliseconds: number): Promise { + return new Promise((resolve) => { + setTimeout(resolve, milliseconds); + }); +} + +async function readExistingRecord( + filePath: string, +): Promise { + let stat: Awaited>; + try { + stat = await fs.lstat(filePath); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return undefined; + throw error; + } + if ( + !stat.isFile() || + stat.isSymbolicLink() || + stat.nlink !== 1 || + (process.platform !== 'win32' && + ((stat.mode & 0o777) !== 0o600 || + (typeof process.getuid === 'function' && + stat.uid !== process.getuid()))) || + stat.size <= 0 || + stat.size > MAX_DISCOVERY_BYTES + ) { + throw new LiveDiscoveryStateError(); + } + let handle: fs.FileHandle; + try { + handle = await fs.open( + filePath, + process.platform === 'win32' + ? fsConstants.O_RDONLY + : fsConstants.O_RDONLY | fsConstants.O_NOFOLLOW, + ); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ELOOP') { + throw new LiveDiscoveryStateError(error); + } + throw error; + } + try { + const handleStat = await handle.stat(); + if ( + !handleStat.isFile() || + handleStat.nlink !== 1 || + handleStat.dev !== stat.dev || + handleStat.ino !== stat.ino || + handleStat.size !== stat.size || + (process.platform !== 'win32' && + ((handleStat.mode & 0o777) !== 0o600 || + (typeof process.getuid === 'function' && + handleStat.uid !== process.getuid()))) + ) { + throw new LiveDiscoveryStateError(); + } + const serialized = await handle.readFile('utf8'); + let parsed: unknown; + try { + parsed = JSON.parse(serialized); + } catch (error) { + throw new LiveDiscoveryStateError(error); + } + if ( + typeof parsed !== 'object' || + parsed === null || + Array.isArray(parsed) + ) { + throw new LiveDiscoveryStateError(); + } + const record = parsed as ExistingLiveDiscoveryRecord; + try { + assertSafeRecord(record); + } catch { + throw new LiveDiscoveryStateError(); + } + return record; + } finally { + await handle.close(); + } +} + +async function inspectDirectory( + directory: string, + requirePrivateMode = true, +): Promise { + const stat = await fs.lstat(directory); + if ( + !stat.isDirectory() || + stat.isSymbolicLink() || + (process.platform !== 'win32' && + ((requirePrivateMode && (stat.mode & 0o777) !== 0o700) || + (typeof process.getuid === 'function' && + stat.uid !== process.getuid()))) + ) { + throw new LiveDiscoveryStateError(); + } + return { directory, dev: stat.dev, ino: stat.ino }; +} + +async function assertDirectoryIdentity( + expected: LiveDiscoveryDirectory, + requirePrivateMode = true, +): Promise { + const current = await inspectDirectory( + expected.directory, + requirePrivateMode, + ); + if (current.dev !== expected.dev || current.ino !== expected.ino) { + throw new LiveDiscoveryStateError(); + } +} + +async function ensureDirectory( + directory: string, + requirePrivateMode: boolean, +): Promise { + try { + return await inspectDirectory(directory, requirePrivateMode); + } catch (error) { + try { + await fs.lstat(directory); + throw error; + } catch (statError) { + if ((statError as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + } + } + + const parent = path.dirname(directory); + if (parent === directory) throw new LiveDiscoveryStateError(); + const parentIdentity = await ensureDirectory(parent, false); + await assertDirectoryIdentity(parentIdentity, false); + let created = false; + try { + await fs.mkdir(directory, { mode: 0o700 }); + created = true; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error; + } + if (created && process.platform !== 'win32') { + await fs.chmod(directory, 0o700); + } + await assertDirectoryIdentity(parentIdentity, false); + return inspectDirectory(directory, requirePrivateMode); +} + +async function assertLockShapeIfPresent(lockPath: string): Promise { + let stat: Awaited>; + try { + stat = await fs.lstat(lockPath); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return; + throw error; + } + if ( + !stat.isDirectory() || + stat.isSymbolicLink() || + (process.platform !== 'win32' && + typeof process.getuid === 'function' && + stat.uid !== process.getuid()) + ) { + throw new LiveDiscoveryStateError(); + } +} + +async function prepareDirectory( + runtimeBaseDir: string, + create = true, +): Promise { + const directory = path.dirname(getLiveDiscoveryPath(runtimeBaseDir)); + let runtimeBaseIdentity: LiveDiscoveryDirectory; + try { + runtimeBaseIdentity = await inspectDirectory(runtimeBaseDir, false); + } catch (error) { + try { + await fs.lstat(runtimeBaseDir); + throw error; + } catch (statError) { + if ((statError as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + } + if (!create) return undefined; + runtimeBaseIdentity = await ensureDirectory(runtimeBaseDir, false); + } + let created = false; + try { + const identity = await inspectDirectory(directory); + await assertDirectoryIdentity(runtimeBaseIdentity, false); + return identity; + } catch (error) { + try { + await fs.lstat(directory); + throw error; + } catch (statError) { + if ((statError as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + } + } + if (!create) return undefined; + await assertDirectoryIdentity(runtimeBaseIdentity, false); + try { + await fs.mkdir(directory, { mode: 0o700 }); + created = true; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error; + } + if (created && process.platform !== 'win32') { + await fs.chmod(directory, 0o700); + } + await assertDirectoryIdentity(runtimeBaseIdentity, false); + return inspectDirectory(directory); +} + +async function lockDirectory(target: LiveDiscoveryDirectory): Promise<{ + release: () => Promise; + assertHealthy: () => void; +}> { + const lockPath = path.join(target.directory, '.daemon.lock'); + await assertLockShapeIfPresent(lockPath); + const lockfile = (await import('proper-lockfile')).default; + let compromised = false; + const release = await lockfile.lock(target.directory, { + ...LOCK_OPTIONS, + lockfilePath: lockPath, + onCompromised: () => { + compromised = true; + }, + }); + try { + await assertLockShapeIfPresent(lockPath); + } catch (error) { + let releaseError: unknown; + await release().catch((candidate: unknown) => { + releaseError = candidate; + }); + throw new LiveDiscoveryStateError( + releaseError + ? new AggregateError( + [error, releaseError], + 'Live discovery lock validation cleanup failed.', + ) + : error, + ); + } + return { + release, + assertHealthy: () => { + if (compromised) throw new LiveDiscoveryStateError(); + }, + }; +} + +async function syncDirectory(directory: string): Promise { + if (process.platform === 'win32') return; + const handle = await fs.open(directory, 'r'); + try { + await handle.sync(); + } finally { + await handle.close(); + } +} + +export async function handoffLiveDiscoveryOwner( + runtimeBaseDir: string, + owner: LiveDiscoveryOwner, + commitOwner: () => Promise, + options: { + isProcessAlive?: (pid: number) => boolean; + wait?: (milliseconds: number) => Promise; + handoffGraceMs?: number; + waitForHandoffGrace?: boolean; + } = {}, +): Promise<{ reclaimed: boolean }> { + const target = await prepareDirectory(runtimeBaseDir, false); + if (!target) { + await commitOwner(); + return { reclaimed: false }; + } + const lock = await lockDirectory(target); + let result: { reclaimed: boolean } | undefined; + let operationError: unknown; + try { + lock.assertHealthy(); + await assertDirectoryIdentity(target); + const filePath = getLiveDiscoveryPath(runtimeBaseDir); + const current = await readExistingRecord(filePath); + if ( + current && + (current.pid !== owner.pid || + current.instanceNonce !== owner.instanceNonce) + ) { + if ( + current.pid === owner.pid || + (options.isProcessAlive ?? processIsAlive)(current.pid) + ) { + throw new LiveDiscoveryOwnerActiveError(current.pid); + } + await commitOwner(); + lock.assertHealthy(); + await assertDirectoryIdentity(target); + const confirmed = await readExistingRecord(filePath); + if ( + !confirmed || + confirmed.pid !== current.pid || + confirmed.instanceNonce !== current.instanceNonce + ) { + throw new LiveDiscoveryStateError(); + } + lock.assertHealthy(); + await assertDirectoryIdentity(target); + await fs.unlink(filePath); + await syncDirectory(target.directory); + lock.assertHealthy(); + result = { reclaimed: true }; + } else { + await commitOwner(); + lock.assertHealthy(); + result = { reclaimed: false }; + } + } catch (error) { + operationError = error; + } + let releaseError: unknown; + await lock.release().catch((error: unknown) => { + releaseError = error; + }); + if (releaseError) { + throw new LiveDiscoveryStateError( + operationError + ? new AggregateError( + [operationError, releaseError], + 'Live discovery handoff cleanup failed.', + ) + : releaseError, + ); + } + if (operationError) throw operationError; + if (result?.reclaimed && options.waitForHandoffGrace !== false) { + await (options.wait ?? delay)( + options.handoffGraceMs ?? DEFAULT_HANDOFF_GRACE_MS, + ); + } + return result!; +} + +export async function writeLiveDiscoveryFile( + runtimeBaseDir: string, + record: LiveDiscoveryRecord, + options: { isProcessAlive?: (pid: number) => boolean } = {}, +): Promise { + assertRecord(record); + const target = await prepareDirectory(runtimeBaseDir); + if (!target) throw new LiveDiscoveryStateError(); + const { directory } = target; + const filePath = getLiveDiscoveryPath(runtimeBaseDir); + const temporaryPath = path.join( + directory, + `.daemon.${process.pid}.${randomUUID()}.tmp`, + ); + const lock = await lockDirectory(target); + let handle: fs.FileHandle | undefined; + let operationError: unknown; + let operationSucceeded = false; + let published = false; + try { + lock.assertHealthy(); + await assertDirectoryIdentity(target); + const current = await readExistingRecord(filePath); + if ( + current && + (current.pid !== record.pid || + current.instanceNonce !== record.instanceNonce) + ) { + if ((options.isProcessAlive ?? processIsAlive)(current.pid)) { + throw new LiveDiscoveryOwnerActiveError(current.pid); + } + throw new LiveDiscoveryStateError(); + } + handle = await fs.open(temporaryPath, 'wx', 0o600); + const serialized = `${JSON.stringify(record)}\n`; + if (Buffer.byteLength(serialized) > MAX_DISCOVERY_BYTES) { + throw new Error('Live discovery record is too large.'); + } + await handle.writeFile(serialized, 'utf8'); + await handle.chmod(0o600); + await handle.sync(); + await handle.close(); + handle = undefined; + lock.assertHealthy(); + await assertDirectoryIdentity(target); + await fs.rename(temporaryPath, filePath); + published = true; + await syncDirectory(directory); + lock.assertHealthy(); + operationSucceeded = true; + } catch (error) { + operationError = error; + } + const cleanupErrors: unknown[] = []; + await handle?.close().catch((error: unknown) => cleanupErrors.push(error)); + await fs.unlink(temporaryPath).catch((error: NodeJS.ErrnoException) => { + if (error.code !== 'ENOENT') cleanupErrors.push(error); + }); + await lock.release().catch((error: unknown) => cleanupErrors.push(error)); + if (!operationSucceeded) { + const cause = + cleanupErrors.length > 0 + ? new AggregateError( + [operationError, ...cleanupErrors], + 'Live discovery publication cleanup failed.', + ) + : operationError; + if (published) throw new LiveDiscoveryPublicationError(cause); + if (cleanupErrors.length > 0) { + throw cause; + } + throw operationError; + } + if (cleanupErrors.length > 0) { + throw new LiveDiscoveryPublicationError( + new AggregateError( + cleanupErrors, + 'Live discovery publication cleanup failed.', + ), + ); + } + return filePath; +} + +export async function removeLiveDiscoveryFile( + runtimeBaseDir: string, + owner: LiveDiscoveryOwner, +): Promise { + if ( + !NONCE_PATTERN.test(owner.instanceNonce) || + !Number.isSafeInteger(owner.pid) || + owner.pid <= 0 + ) { + throw new LiveDiscoveryStateError(); + } + const target = await prepareDirectory(runtimeBaseDir, false); + if (!target) return false; + const filePath = getLiveDiscoveryPath(runtimeBaseDir); + const lock = await lockDirectory(target); + try { + lock.assertHealthy(); + await assertDirectoryIdentity(target); + const current = await readExistingRecord(filePath); + if (!current) return false; + if ( + current.instanceNonce !== owner.instanceNonce || + current.pid !== owner.pid + ) { + throw new LiveDiscoveryStateError(); + } + lock.assertHealthy(); + await assertDirectoryIdentity(target); + await fs.unlink(filePath); + await syncDirectory(target.directory); + lock.assertHealthy(); + return true; + } finally { + await lock.release(); + } +} diff --git a/packages/qwen-live/src/host/live-host-coordinator.test.ts b/packages/qwen-live/src/host/live-host-coordinator.test.ts new file mode 100644 index 00000000000..3f4ff142470 --- /dev/null +++ b/packages/qwen-live/src/host/live-host-coordinator.test.ts @@ -0,0 +1,1206 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { EventEmitter } from 'node:events'; +import { WebSocket } from 'ws'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { + LiveHostCoordinator, + LiveUnavailableError, +} from './live-host-coordinator.js'; +import { + LIVE_HOST_BUNDLE_ID, + LIVE_HOST_PROTOCOL_VERSION, + LIVE_INPUT_AUDIO_EPOCH_BYTES, + type LiveDaemonMessage, + type LiveHostHello, +} from './types.js'; + +class FakeSocket extends EventEmitter { + readyState: number = WebSocket.OPEN; + bufferedAmount = 0; + readonly sent: Array = []; + closeCode?: number; + + send(data: string | Uint8Array): void { + this.sent.push(data); + } + + close(code?: number): void { + this.closeCode = code; + this.readyState = WebSocket.CLOSED; + this.emit('close'); + } + + receive(message: unknown): void { + const data = + typeof message === 'string' + ? Buffer.from(message) + : Buffer.from(JSON.stringify(message)); + this.emit('message', data, false); + } + + receiveAudio(epoch: number, bytes: readonly number[]): void { + const frame = Buffer.alloc(LIVE_INPUT_AUDIO_EPOCH_BYTES + bytes.length); + frame.writeBigUInt64BE(BigInt(epoch), 0); + Buffer.from(bytes).copy(frame, LIVE_INPUT_AUDIO_EPOCH_BYTES); + this.emit('message', frame, true); + } + + receiveRawAudio(bytes: readonly number[]): void { + this.emit('message', Buffer.from(bytes), true); + } + + messages(): LiveDaemonMessage[] { + return this.sent + .filter((value): value is string => typeof value === 'string') + .map((value) => JSON.parse(value) as LiveDaemonMessage); + } +} + +const coordinators: LiveHostCoordinator[] = []; + +function readyHello(overrides: Partial = {}): LiveHostHello { + return { + type: 'host.hello', + protocolVersion: LIVE_HOST_PROTOCOL_VERSION, + hostVersion: '1.0.0', + bundleId: LIVE_HOST_BUNDLE_ID, + instanceNonce: 'host_instance_nonce_0001', + permissions: { + microphone: 'granted', + accessibility: 'granted', + screenRecording: 'granted', + }, + selfChecks: { + audioInput: true, + audioOutput: true, + globalShortcut: true, + appshot: true, + }, + ...overrides, + }; +} + +function coordinator( + options: Partial[0]> = {}, +): LiveHostCoordinator { + const value = new LiveHostCoordinator({ + daemonInstanceNonce: 'daemon_instance_nonce_0001', + getProviderReadiness: () => ({ state: 'ready' }), + ...options, + }); + value.setAppshotReadiness({ state: 'ready' }); + coordinators.push(value); + return value; +} + +function connectReady( + value: LiveHostCoordinator, + hello = readyHello(), +): FakeSocket { + const socket = new FakeSocket(); + value.attachHost(socket as unknown as WebSocket, value.daemonInstanceNonce); + socket.receive(hello); + return socket; +} + +afterEach(() => { + for (const value of coordinators.splice(0)) value.dispose(); + vi.useRealTimers(); +}); + +describe('LiveHostCoordinator', () => { + it('routes one correlated Appshot only for the active Live session', async () => { + const value = coordinator(); + const socket = connectReady(value); + const call = value.start('resume'); + value.setCoordinator(call.epoch, { + workspaceCwd: '/conversations/live-1', + sessionId: 'coordinator-1', + }); + + await expect(value.captureScreenContext('worker-1')).rejects.toThrow( + 'active Live session', + ); + const capture = value.captureScreenContext('coordinator-1'); + const request = socket + .messages() + .find((message) => message.type === 'host.capture_screen_context'); + expect(request).toMatchObject({ + type: 'host.capture_screen_context', + epoch: call.epoch, + }); + if (!request || request.type !== 'host.capture_screen_context') { + throw new Error('Missing Appshot request'); + } + socket.receive({ + type: 'host.screen_context_result', + requestId: request.requestId, + success: true, + appName: 'Google Chrome', + windowTitle: 'LIVE_APP_A', + accessibilityText: 'AXWindow LIVE_APP_A', + screenshotPath: '/private/tmp/qwen-live-appshot/test.png', + }); + + await expect(capture).resolves.toEqual({ + appName: 'Google Chrome', + windowTitle: 'LIVE_APP_A', + accessibilityText: 'AXWindow LIVE_APP_A', + screenshotPath: '/private/tmp/qwen-live-appshot/test.png', + }); + value.stop(); + }); + + it('projects a WebShell target only while the coordinator awaits permission', () => { + const value = coordinator(); + connectReady(value); + const call = value.start('resume'); + value.setCoordinator(call.epoch, { + workspaceCwd: '/conversations/live-1', + workspaceId: 'conversations-workspace', + sessionId: 'coordinator-1', + }); + + expect(value.setPendingPermission(call.epoch, true)).toBe(true); + expect(value.getStatus().pendingPermission).toEqual({ + workspaceId: 'conversations-workspace', + sessionId: 'coordinator-1', + }); + + expect(value.setPendingPermission(call.epoch, false)).toBe(true); + expect(value.getStatus().pendingPermission).toBeUndefined(); + expect(value.setPendingPermission(call.epoch + 1, true)).toBe(false); + }); + + it('lets the active Live session finish Appshot during stop drain', async () => { + let finishStop: (() => void) | undefined; + const value = coordinator({ + handlers: { + onStop: () => + new Promise((resolve) => { + finishStop = resolve; + }), + }, + }); + const socket = connectReady(value); + const call = value.start('resume'); + + expect(value.stop()).toMatchObject({ + state: 'stopping', + callId: call.callId, + }); + expect( + value.setCoordinator(call.epoch, { + workspaceCwd: '/conversations/live-1', + sessionId: 'coordinator-1', + }), + ).toBe(true); + + const capture = value.captureScreenContext('coordinator-1'); + const request = socket + .messages() + .find((message) => message.type === 'host.capture_screen_context'); + if (!request || request.type !== 'host.capture_screen_context') { + throw new Error('Missing Appshot request'); + } + socket.receive({ + type: 'host.screen_context_result', + requestId: request.requestId, + success: true, + appName: 'TextEdit', + accessibilityText: 'APPSHOT-MARKER-AMBER-4827', + screenshotPath: '/private/tmp/qwen-live-appshot/test.png', + }); + + await expect(capture).resolves.toMatchObject({ + appName: 'TextEdit', + accessibilityText: 'APPSHOT-MARKER-AMBER-4827', + }); + finishStop?.(); + await vi.waitFor(() => { + expect(value.getStatus()).toMatchObject({ state: 'idle' }); + }); + }); + + it('bounds a Host capture that never answers', async () => { + vi.useFakeTimers(); + const value = coordinator({ appshotTimeoutMs: 100 }); + connectReady(value); + const call = value.start('resume'); + value.setCoordinator(call.epoch, { + workspaceCwd: '/conversations/live-1', + sessionId: 'coordinator-1', + }); + const capture = value.captureScreenContext('coordinator-1'); + const settled = capture.catch((error: unknown) => error); + + await vi.advanceTimersByTimeAsync(100); + + await expect(settled).resolves.toMatchObject({ + message: expect.stringContaining('timed out'), + }); + value.stop(); + }); + + it('fails closed until Appshot has been verified', () => { + const value = new LiveHostCoordinator({ + daemonInstanceNonce: 'daemon_instance_nonce_0002', + getProviderReadiness: () => ({ state: 'ready' }), + shortcut: 'Command+Shift+L', + }); + coordinators.push(value); + connectReady(value); + + expect(value.getStatus()).toMatchObject({ + available: false, + blocker: 'appshot', + shortcut: 'Command+Shift+L', + requirements: { appshot: 'unavailable' }, + }); + }); + + it('commits a shortcut only after the Host confirms registration', async () => { + const value = coordinator(); + const socket = connectReady(value); + + const update = value.setShortcut('Command+Shift+E'); + const request = socket + .messages() + .find((message) => message.type === 'host.set_shortcut'); + expect(request).toMatchObject({ + type: 'host.set_shortcut', + shortcut: 'Command+Shift+E', + }); + expect(value.getStatus().shortcut).toBe('Command+E'); + if (!request || request.type !== 'host.set_shortcut') { + throw new Error('Missing shortcut request'); + } + socket.receive({ + type: 'host.shortcut_result', + requestId: request.requestId, + shortcut: request.shortcut, + success: true, + }); + + await expect(update).resolves.toMatchObject({ + shortcut: 'Command+Shift+E', + }); + expect(value.getStatus().shortcut).toBe('Command+Shift+E'); + }); + + it('keeps the previous shortcut when the Host rejects a conflict', async () => { + const value = coordinator(); + const socket = connectReady(value); + + const update = value.setShortcut('Command+Shift+E'); + const request = socket + .messages() + .find((message) => message.type === 'host.set_shortcut'); + if (!request || request.type !== 'host.set_shortcut') { + throw new Error('Missing shortcut request'); + } + socket.receive({ + type: 'host.shortcut_result', + requestId: request.requestId, + shortcut: request.shortcut, + success: false, + error: 'That shortcut is already in use.', + }); + + await expect(update).rejects.toThrow('already in use'); + expect(value.getStatus().shortcut).toBe('Command+E'); + }); + + it('can turn the global shortcut off without disabling Live', async () => { + const value = coordinator(); + const socket = connectReady(value); + + const update = value.setShortcut(''); + const request = socket + .messages() + .find((message) => message.type === 'host.set_shortcut'); + if (!request || request.type !== 'host.set_shortcut') { + throw new Error('Missing shortcut request'); + } + socket.receive({ + type: 'host.shortcut_result', + requestId: request.requestId, + shortcut: '', + success: true, + }); + + await expect(update).resolves.toMatchObject({ + available: true, + shortcut: '', + }); + }); + + it('requires the discovery nonce before accepting a Host', () => { + const value = coordinator(); + const socket = new FakeSocket(); + + value.attachHost(socket as unknown as WebSocket, 'wrong_nonce_value_0000'); + + expect(socket.closeCode).toBe(4003); + expect(value.getStatus()).toMatchObject({ + available: false, + blocker: 'host_missing', + }); + }); + + it('welcomes one compatible, fully-authorized Host', () => { + const value = coordinator(); + const socket = connectReady(value); + + expect(value.getStatus()).toMatchObject({ + available: true, + state: 'idle', + host: { + version: '1.0.0', + protocolVersion: LIVE_HOST_PROTOCOL_VERSION, + }, + requirements: { + host: 'ready', + microphone: 'ready', + accessibility: 'ready', + screenRecording: 'ready', + audioInput: 'ready', + audioOutput: 'ready', + globalShortcut: 'ready', + appshot: 'ready', + provider: 'ready', + }, + }); + expect(socket.messages().map((message) => message.type)).toEqual([ + 'host.welcome', + 'host.state', + ]); + + const duplicate = new FakeSocket(); + value.attachHost( + duplicate as unknown as WebSocket, + value.daemonInstanceNonce, + ); + expect(duplicate.closeCode).toBe(4009); + }); + + it('never sends WebShell session locators to the native Host', () => { + const value = coordinator(); + const socket = connectReady(value); + const call = value.start('new'); + value.setCoordinator(call.epoch, { + workspaceCwd: '/private/conversations/coordinator', + sessionId: 'coordinator-session', + }); + value.setWorkers(call.epoch, [ + { + workspaceCwd: '/private/conversations/worker', + sessionId: 'worker-session', + }, + ]); + + expect(value.getStatus()).not.toHaveProperty('coordinator'); + expect(value.getStatus()).not.toHaveProperty('workers'); + expect(value.isActiveSession('coordinator-session')).toBe(true); + expect(value.isActiveSession('worker-session')).toBe(true); + expect(value.isActiveSession('unrelated-session')).toBe(false); + for (const message of socket.messages()) { + if (message.type !== 'host.welcome' && message.type !== 'host.state') { + continue; + } + expect(message.status).not.toHaveProperty('coordinator'); + expect(message.status).not.toHaveProperty('workers'); + expect(JSON.stringify(message)).not.toContain('/private/conversations'); + } + }); + + it('hard-gates start on every permission and self-check', () => { + const value = coordinator(); + connectReady( + value, + readyHello({ + permissions: { + ...readyHello().permissions, + screenRecording: 'denied', + }, + }), + ); + + expect(() => value.start('resume')).toThrow(LiveUnavailableError); + expect(value.getStatus()).toMatchObject({ + available: false, + state: 'unavailable', + blocker: 'screen_recording_permission', + requirements: { screenRecording: 'denied' }, + }); + }); + + it('hard-gates start when the built-in Appshot channel is unavailable', () => { + const value = coordinator(); + connectReady(value); + + value.setAppshotReadiness({ + state: 'unavailable', + message: 'The built-in Appshot channel is unavailable.', + }); + + expect(() => value.start('resume')).toThrow(LiveUnavailableError); + expect(value.getStatus()).toMatchObject({ + available: false, + state: 'unavailable', + blocker: 'appshot', + message: 'The built-in Appshot channel is unavailable.', + requirements: { appshot: 'unavailable' }, + }); + }); + + it('owns one call epoch and ignores stale updates and actions', () => { + const starts = vi.fn(); + const stops = vi.fn(); + const value = coordinator({ handlers: { onStart: starts, onStop: stops } }); + const socket = connectReady(value); + + const first = value.start('resume'); + expect(value.start('resume').callId).toBe(first.callId); + const second = value.start('new'); + + expect(second.callId).not.toBe(first.callId); + expect(second.epoch).toBeGreaterThan(first.epoch); + expect(starts).toHaveBeenCalledTimes(2); + expect(stops).toHaveBeenCalledWith({ + epoch: first.epoch, + callId: first.callId, + }); + expect(value.setCallState(first.epoch, 'speaking')).toBe(false); + + socket.receive({ + type: 'host.action', + action: 'stop', + epoch: first.epoch, + }); + expect(value.getStatus().callId).toBe(second.callId); + expect(socket.messages()).toContainEqual({ + type: 'host.error', + code: 'stale_epoch', + message: 'The Live action epoch is stale.', + }); + + value.stop(); + socket.receive({ + type: 'host.action', + action: 'new', + epoch: second.epoch, + }); + expect(value.getStatus().state).toBe('idle'); + }); + + it('keeps the stopping call epoch until the session drain succeeds', async () => { + let finishStop: (() => void) | undefined; + const onStop = vi.fn( + () => + new Promise((resolve) => { + finishStop = resolve; + }), + ); + const value = coordinator({ handlers: { onStop } }); + connectReady(value); + const call = value.start('resume'); + + expect(value.stop()).toMatchObject({ + state: 'stopping', + callId: call.callId, + }); + expect(value.setTranscript(call.epoch, 'late final transcript')).toBe(true); + expect(value.getStatus()).toMatchObject({ + state: 'stopping', + callId: call.callId, + transcript: 'late final transcript', + }); + + finishStop?.(); + await vi.waitFor(() => { + expect(value.getStatus()).toMatchObject({ state: 'idle' }); + expect(value.getStatus().callId).toBeUndefined(); + }); + }); + + it('projects user transcript, assistant caption, and task status separately', () => { + const value = coordinator(); + const socket = connectReady(value); + const call = value.start('resume'); + + expect(value.setTranscript(call.epoch, '检查当前页面')).toBe(true); + expect(value.setCaption(call.epoch, '当前页面是文档编辑器。')).toBe(true); + expect(value.setStatusText(call.epoch, 'Reading screen…')).toBe(true); + + expect(value.getStatus()).toMatchObject({ + transcript: '检查当前页面', + caption: '当前页面是文档编辑器。', + statusText: 'Reading screen…', + }); + expect(socket.messages().at(-1)).toMatchObject({ + type: 'host.state', + status: { + transcript: '检查当前页面', + caption: '当前页面是文档编辑器。', + statusText: 'Reading screen…', + }, + }); + }); + + it('starts a replacement only after the exact pending input is persisted', async () => { + let finishPersistence!: () => void; + const persisted = new Promise((resolve) => { + finishPersistence = resolve; + }); + const lifecycle: string[] = []; + const onStart = vi.fn((call: { mode: 'resume' | 'new' }) => { + lifecycle.push(`start:${call.mode}`); + }); + const onStop = vi.fn(async () => { + lifecycle.push('stop:requested'); + await persisted; + lifecycle.push('persist:exact-final'); + }); + const value = coordinator({ handlers: { onStart, onStop } }); + connectReady(value); + const first = value.start('resume'); + + const pending = value.start('new'); + + expect(pending).toMatchObject({ + epoch: first.epoch, + callId: first.callId, + status: { state: 'stopping', callId: first.callId }, + }); + expect(onStart).toHaveBeenCalledOnce(); + expect(lifecycle).toEqual(['start:resume', 'stop:requested']); + + finishPersistence(); + await vi.waitFor(() => expect(onStart).toHaveBeenCalledTimes(2)); + + expect(lifecycle).toEqual([ + 'start:resume', + 'stop:requested', + 'persist:exact-final', + 'start:new', + ]); + expect(value.getStatus()).toMatchObject({ state: 'starting' }); + expect(value.getStatus().callId).not.toBe(first.callId); + }); + + it('does not rotate when persistence fails during replacement', async () => { + const onStart = vi.fn(); + const value = coordinator({ + handlers: { + onStart, + onStop: async () => ({ + error: 'Exact final transcript was not saved.', + }), + }, + }); + connectReady(value); + value.start('resume'); + + value.start('new'); + + await vi.waitFor(() => { + expect(value.getStatus()).toMatchObject({ + state: 'error', + message: 'Exact final transcript was not saved.', + }); + }); + expect(onStart).toHaveBeenCalledOnce(); + }); + + it('lets an explicit stop cancel a pending replacement', async () => { + let finishStop!: () => void; + const onStart = vi.fn(); + const value = coordinator({ + handlers: { + onStart, + onStop: () => + new Promise((resolve) => { + finishStop = resolve; + }), + }, + }); + connectReady(value); + value.start('resume'); + value.start('new'); + + value.stop(); + finishStop(); + + await vi.waitFor(() => + expect(value.getStatus()).toMatchObject({ state: 'idle' }), + ); + expect(onStart).toHaveBeenCalledOnce(); + }); + + it('publishes a visible error when the session drain cannot be confirmed', async () => { + let finishStop: ((outcome: { error: string }) => void) | undefined; + const onStop = vi.fn( + () => + new Promise<{ error: string }>((resolve) => { + finishStop = resolve; + }), + ); + const value = coordinator({ handlers: { onStop } }); + connectReady(value); + value.start('resume'); + value.stop(); + + finishStop?.({ error: 'The final spoken input was not persisted.' }); + await vi.waitFor(() => { + expect(value.getStatus()).toMatchObject({ + state: 'error', + message: 'The final spoken input was not persisted.', + }); + }); + }); + + it('rejects the removed permission and session actions', () => { + const removedActions = [ + { + type: 'host.action', + action: 'request_permission', + permission: 'microphone', + }, + { + type: 'host.action', + action: 'open_session', + locator: { workspaceCwd: '/work/one', sessionId: 'session-1' }, + }, + ]; + + for (const action of removedActions) { + const value = coordinator(); + const socket = connectReady(value); + + socket.receive(action); + + expect(socket.closeCode).toBe(1002); + expect(socket.messages()).toContainEqual({ + type: 'host.error', + code: 'invalid_message', + message: 'Invalid Live Host message.', + }); + } + }); + + it('contains a synchronous start-handler failure in the call state', async () => { + const value = coordinator({ + handlers: { + onStart: () => { + throw new Error('start failed'); + }, + }, + }); + connectReady(value); + + expect(() => value.start('resume')).not.toThrow(); + await vi.waitFor(() => { + expect(value.getStatus()).toMatchObject({ + available: true, + state: 'error', + message: 'Live Voice failed to start.', + }); + }); + }); + + it('forwards bounded PCM only for an active, unmuted call', () => { + const onInputAudio = vi.fn(); + const value = coordinator({ handlers: { onInputAudio } }); + const socket = connectReady(value); + const call = value.start('resume'); + + socket.receiveAudio(call.epoch, [0, 0, 1, 0]); + expect(onInputAudio).toHaveBeenCalledWith({ + epoch: call.epoch, + callId: call.callId, + pcm16: Buffer.from([0, 0, 1, 0]), + }); + + value.setMute({ inputMuted: true, outputMuted: true }); + socket.receiveAudio(call.epoch, [2, 0]); + expect(onInputAudio).toHaveBeenCalledTimes(1); + const sentBeforeMutedOutput = socket.sent.length; + expect(value.sendOutputAudio(call.epoch, Buffer.from([0, 0]))).toBe(true); + expect(value.sendOutputAudio(call.epoch, Buffer.from([1, 0]))).toBe(true); + expect(socket.sent).toHaveLength(sentBeforeMutedOutput); + expect(value.getStatus()).toMatchObject({ + callId: call.callId, + outputMuted: true, + }); + expect(socket.messages()).toContainEqual({ + type: 'host.clear_output', + epoch: call.epoch, + }); + }); + + it('fails the call when the provider audio path rejects a frame', () => { + const value = coordinator({ + handlers: { onInputAudio: () => false }, + }); + const socket = connectReady(value); + const call = value.start('resume'); + + socket.receiveAudio(call.epoch, [0, 0]); + + expect(value.getStatus()).toMatchObject({ + state: 'error', + message: 'Live Voice audio transport dropped input.', + }); + }); + + it('drops same-epoch audio while stop is draining', async () => { + let finishStop: (() => void) | undefined; + const onInputAudio = vi.fn(() => false); + const onStop = vi.fn( + () => + new Promise((resolve) => { + finishStop = resolve; + }), + ); + const value = coordinator({ handlers: { onInputAudio, onStop } }); + const socket = connectReady(value); + const call = value.start('resume'); + + value.stop(); + socket.receiveAudio(call.epoch, [0, 0]); + + expect(onInputAudio).not.toHaveBeenCalled(); + expect(onStop).toHaveBeenCalledOnce(); + expect(value.getStatus()).toMatchObject({ + state: 'stopping', + callId: call.callId, + }); + expect(value.getStatus().message).toBeUndefined(); + + finishStop?.(); + await vi.waitFor(() => { + expect(value.getStatus()).toMatchObject({ state: 'idle' }); + }); + expect(value.getStatus().callId).toBeUndefined(); + expect(value.getStatus().message).toBeUndefined(); + expect(value.start('resume').epoch).toBeGreaterThan(call.epoch); + }); + + it('drops audio from the previous epoch after starting a new call', () => { + const onInputAudio = vi.fn(); + const value = coordinator({ handlers: { onInputAudio } }); + const socket = connectReady(value); + const first = value.start('resume'); + const second = value.start('new'); + + socket.receiveAudio(first.epoch, [1, 0]); + expect(onInputAudio).not.toHaveBeenCalled(); + + socket.receiveAudio(second.epoch, [2, 0]); + expect(onInputAudio).toHaveBeenCalledOnce(); + expect(onInputAudio).toHaveBeenCalledWith({ + epoch: second.epoch, + callId: second.callId, + pcm16: Buffer.from([2, 0]), + }); + }); + + it('fails the call instead of crashing when onInputAudio stops and throws', () => { + const onStop = vi.fn(); + const ref: { current: LiveHostCoordinator | undefined } = { + current: undefined, + }; + const onInputAudio = vi.fn(() => { + ref.current!.stop(); + throw new Error('handler bug'); + }); + const value = coordinator({ handlers: { onInputAudio, onStop } }); + ref.current = value; + const socket = connectReady(value); + const call = value.start('resume'); + + socket.receiveAudio(call.epoch, [1, 0]); + + expect(onInputAudio).toHaveBeenCalledOnce(); + expect(onStop).toHaveBeenCalledOnce(); + expect(socket.closeCode).toBeUndefined(); + }); + + it('rejects binary input without the protocol-v2 epoch header', () => { + const value = coordinator(); + const socket = connectReady(value); + value.start('resume'); + + socket.receiveRawAudio([1, 0]); + + expect(socket.closeCode).toBe(1009); + }); + + it('fails closed and stops the call when provider readiness is lost', () => { + const onStop = vi.fn(); + const value = coordinator({ handlers: { onStop } }); + connectReady(value); + const call = value.start('resume'); + + value.setProviderReachability({ + state: 'unavailable', + blocker: 'provider_unreachable', + }); + + expect(value.getStatus()).toMatchObject({ + available: false, + state: 'unavailable', + blocker: 'provider_unreachable', + }); + expect(onStop).toHaveBeenCalledWith({ + epoch: call.epoch, + callId: call.callId, + }); + }); + + it('retains session ownership while readiness-loss persistence drains', async () => { + let finishStop: (() => void) | undefined; + const stopPending = new Promise((resolve) => { + finishStop = resolve; + }); + const value = coordinator({ + handlers: { onStop: () => stopPending }, + }); + connectReady(value); + const call = value.start('resume'); + value.setCoordinator(call.epoch, { + workspaceCwd: '/conversations/live-1', + sessionId: 'session-live-1', + }); + + value.setProviderReachability({ + state: 'unavailable', + blocker: 'provider_unreachable', + }); + + expect(value.isActiveSession('session-live-1')).toBe(true); + expect(value.getStatus()).toMatchObject({ + available: false, + state: 'unavailable', + callId: call.callId, + }); + + finishStop?.(); + await vi.waitFor(() => { + expect(value.isActiveSession('session-live-1')).toBe(false); + }); + }); + + it('keeps the active call while provider readiness is checking but rejects a new start', () => { + const onStop = vi.fn(); + const value = coordinator({ handlers: { onStop } }); + connectReady(value); + const call = value.start('resume'); + value.setCoordinator(call.epoch, { + workspaceCwd: '/conversations/live-1', + sessionId: 'session-live-1', + }); + + value.setProviderReachability({ state: 'checking' }); + + expect(value.getStatus()).toMatchObject({ + available: false, + state: 'starting', + callId: call.callId, + requirements: { provider: 'checking' }, + }); + expect(value.getStatus().blocker).toBeUndefined(); + expect(onStop).not.toHaveBeenCalled(); + expect(() => value.start('new')).toThrow(LiveUnavailableError); + expect(onStop).not.toHaveBeenCalled(); + + value.setProviderReachability(undefined); + value.stop(); + }); + + it('stops an active call when Appshot is lost while the provider is checking', () => { + const onStop = vi.fn(); + const value = coordinator({ handlers: { onStop } }); + connectReady(value); + const call = value.start('resume'); + value.setProviderReachability({ state: 'checking' }); + + value.setAppshotReadiness({ + state: 'unavailable', + message: 'Appshot tools became unavailable.', + }); + + expect(value.getStatus()).toMatchObject({ + available: false, + state: 'unavailable', + blocker: 'appshot', + message: 'Appshot tools became unavailable.', + requirements: { provider: 'checking', appshot: 'unavailable' }, + }); + expect(value.getStatus().callId).toBeUndefined(); + expect(onStop).toHaveBeenCalledOnce(); + expect(onStop).toHaveBeenCalledWith({ + epoch: call.epoch, + callId: call.callId, + }); + }); + + it('stops an active call when a permission is lost while the provider is checking', () => { + const onStop = vi.fn(); + const value = coordinator({ handlers: { onStop } }); + const socket = connectReady(value); + const call = value.start('resume'); + value.setProviderReachability({ state: 'checking' }); + + socket.receive( + readyHello({ + permissions: { + ...readyHello().permissions, + screenRecording: 'denied', + }, + }), + ); + + expect(value.getStatus()).toMatchObject({ + available: false, + state: 'unavailable', + blocker: 'screen_recording_permission', + requirements: { provider: 'checking', screenRecording: 'denied' }, + }); + expect(value.getStatus().callId).toBeUndefined(); + expect(onStop).toHaveBeenCalledOnce(); + expect(onStop).toHaveBeenCalledWith({ + epoch: call.epoch, + callId: call.callId, + }); + }); + + it('stops exactly once when readiness changes during an explicit stop', () => { + let providerReady = true; + const onStop = vi.fn(); + const value = coordinator({ + getProviderReadiness: () => + providerReady + ? { state: 'ready' } + : { state: 'unavailable', blocker: 'provider_unreachable' }, + handlers: { onStop }, + }); + connectReady(value); + value.start('resume'); + + providerReady = false; + value.stop(); + + expect(onStop).toHaveBeenCalledOnce(); + expect(value.getStatus()).toMatchObject({ + available: false, + blocker: 'provider_unreachable', + }); + }); + + it('expires a Host that misses the application heartbeat', async () => { + vi.useFakeTimers(); + let now = 0; + const onStop = vi.fn(); + const value = coordinator({ + now: () => now, + heartbeatIntervalMs: 5, + heartbeatTimeoutMs: 10, + handlers: { onStop }, + }); + const socket = connectReady(value); + value.start('resume'); + + now = 20; + await vi.advanceTimersByTimeAsync(5); + + expect(socket.closeCode).toBe(4008); + expect(value.getStatus()).toMatchObject({ + available: false, + blocker: 'host_disconnected', + }); + expect(onStop).toHaveBeenCalledOnce(); + }); + + it('stops the active call before disconnecting the Host on disable', async () => { + let finishStop: (() => void) | undefined; + const value = coordinator({ + handlers: { + onStop: () => + new Promise((resolve) => { + finishStop = resolve; + }), + }, + }); + const socket = connectReady(value); + value.start('resume'); + + const deactivating = value.deactivate(); + expect(socket.closeCode).toBeUndefined(); + expect(value.getStatus().state).toBe('stopping'); + + finishStop?.(); + await deactivating; + expect(socket.closeCode).toBe(1001); + expect(value.getStatus().callId).toBeUndefined(); + }); + + it('applies a configured shortcut before a Host connects', () => { + const value = coordinator(); + expect(value.setConfiguredShortcut('Command+K').shortcut).toBe('Command+K'); + }); + + it('accepts a maximum-length CJK accessibility dump without closing the socket', async () => { + const value = coordinator(); + const socket = connectReady(value); + const call = value.start('resume'); + value.setCoordinator(call.epoch, { + workspaceCwd: '/conversations/live-1', + sessionId: 'coordinator-1', + }); + + const capture = value.captureScreenContext('coordinator-1'); + const request = socket + .messages() + .find((message) => message.type === 'host.capture_screen_context'); + if (!request || request.type !== 'host.capture_screen_context') { + throw new Error('Missing Appshot request'); + } + // 32,000 CJK chars are conformant per MAX_APPSHOT_TEXT_LENGTH but weigh + // ~96 KiB in UTF-8: the frame cap must admit them. + const accessibilityText = '中'.repeat(32_000); + socket.receive({ + type: 'host.screen_context_result', + requestId: request.requestId, + success: true, + appName: '微信', + accessibilityText, + screenshotPath: '/private/tmp/qwen-live-appshot/test.png', + }); + + await expect(capture).resolves.toMatchObject({ accessibilityText }); + expect(socket.closeCode).toBeUndefined(); + expect(pendingAppshotCount(value)).toBe(0); + }); + + it('ignores host-initiated starts while deactivating', async () => { + let finishStop: (() => void) | undefined; + const onStart = vi.fn(); + const value = coordinator({ + handlers: { + onStart, + onStop: () => + new Promise((resolve) => { + finishStop = resolve; + }), + }, + }); + const socket = connectReady(value); + const call = value.start('resume'); + expect(onStart).toHaveBeenCalledTimes(1); + + const deactivating = value.deactivate(); + // The stopping call's epoch still passes the epoch gate, but the start + // must be ignored during the deactivation drain. + socket.receive({ type: 'host.action', action: 'new', epoch: call.epoch }); + finishStop?.(); + await deactivating; + + expect(value.getStatus().callId).toBeUndefined(); + expect(onStart).toHaveBeenCalledTimes(1); + }); + + it('lets a host-initiated start work again after the Host reattaches', async () => { + const onStart = vi.fn(); + const value = coordinator({ handlers: { onStart } }); + connectReady(value); + await value.deactivate(); + + const socket = connectReady(value); + socket.receive({ type: 'host.action', action: 'new' }); + expect(onStart).toHaveBeenCalledTimes(1); + }); + + it('removes a timed-out Appshot from the pending map', async () => { + vi.useFakeTimers(); + const value = coordinator({ appshotTimeoutMs: 50 }); + connectReady(value); + const call = value.start('resume'); + value.setCoordinator(call.epoch, { + workspaceCwd: '/conversations/live-1', + sessionId: 'coordinator-1', + }); + + const capture = value.captureScreenContext('coordinator-1'); + // Attach the rejection handler BEFORE advancing the clock: the + // timeout fires synchronously inside advanceTimersByTimeAsync, and an + // unhandled rejection at that instant fails the whole CI run (vitest + // counts it as an error even though every test passes). + // The handler must exist before the clock advances (see comment above); + // awaited on the line after advanceTimersByTimeAsync, which the lint + // rule cannot see across the assignment. + // eslint-disable-next-line vitest/valid-expect + const rejection = expect(capture).rejects.toThrow('timed out'); + expect(pendingAppshotCount(value)).toBe(1); + await vi.advanceTimersByTimeAsync(51); + + await rejection; + expect(pendingAppshotCount(value)).toBe(0); + }); + + it('drops a rejected Appshot from the pending map when the call stops', async () => { + const value = coordinator(); + connectReady(value); + const call = value.start('resume'); + value.setCoordinator(call.epoch, { + workspaceCwd: '/conversations/live-1', + sessionId: 'coordinator-1', + }); + + const capture = value.captureScreenContext('coordinator-1'); + expect(pendingAppshotCount(value)).toBe(1); + value.stop(); + + await expect(capture).rejects.toThrow('ended before Appshot completed'); + expect(pendingAppshotCount(value)).toBe(0); + }); + + it('clears resolved inactive waiters instead of retaining them', async () => { + let finishStop: (() => void) | undefined; + const value = coordinator({ + handlers: { + onStop: () => + new Promise((resolve) => { + finishStop = resolve; + }), + }, + }); + connectReady(value); + value.start('resume'); + + const deactivating = value.deactivate(); + expect(inactiveWaiterCount(value)).toBe(1); + finishStop?.(); + await deactivating; + + expect(inactiveWaiterCount(value)).toBe(0); + }); +}); + +/** Reach into the private pending-Appshot map to pin its cleanup paths. */ +function pendingAppshotCount(value: LiveHostCoordinator): number { + return (value as unknown as { pendingAppshots: Map }) + .pendingAppshots.size; +} + +/** Reach into the private inactive-waiter set to pin notifyInactive. */ +function inactiveWaiterCount(value: LiveHostCoordinator): number { + return (value as unknown as { inactiveWaiters: Set<() => void> }) + .inactiveWaiters.size; +} diff --git a/packages/qwen-live/src/host/live-host-coordinator.ts b/packages/qwen-live/src/host/live-host-coordinator.ts new file mode 100644 index 00000000000..6c68b749998 --- /dev/null +++ b/packages/qwen-live/src/host/live-host-coordinator.ts @@ -0,0 +1,1377 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { randomUUID, timingSafeEqual } from 'node:crypto'; +import { WebSocket, type RawData } from 'ws'; +import { + LIVE_HOST_BUNDLE_ID, + LIVE_HOST_PROTOCOL_VERSION, + LIVE_INPUT_AUDIO_EPOCH_BYTES, + type LiveAppshotReadiness, + type LiveDaemonMessage, + type LiveHostAction, + type LiveHostHello, + type LiveHostShortcutResult, + type LiveHostScreenContextResult, + type LiveHostStatus, + type LiveHostMessage, + type LiveMuteUpdate, + type LivePermissionState, + type LiveProviderReadiness, + type LiveSessionLocator, + type LiveState, + type LiveStatus, +} from './types.js'; + +const DEFAULT_HELLO_TIMEOUT_MS = 5_000; +const DEFAULT_HEARTBEAT_INTERVAL_MS = 5_000; +const DEFAULT_HEARTBEAT_TIMEOUT_MS = 15_000; +const DEFAULT_SHORTCUT_TIMEOUT_MS = 5_000; +// The frame cap must admit every message the per-field caps below allow. The +// worst conformant case is a host.screen_context_result: requestId (128) + +// appName (512) + windowTitle (2,048) + accessibilityText (32,000) + +// screenshotPath (4,096) ≈ 38,784 UTF-16 units, each up to 6 UTF-8 bytes once +// JSON-escaped (≈ 227 KiB), plus JSON syntax overhead — 256 KiB covers it. +const MAX_HOST_TEXT_BYTES = 256 * 1024; +const MAX_HOST_AUDIO_BYTES = 64 * 1024; +const MAX_HOST_AUDIO_WIRE_BYTES = + LIVE_INPUT_AUDIO_EPOCH_BYTES + MAX_HOST_AUDIO_BYTES; +const MAX_DAEMON_AUDIO_BYTES = 256 * 1024; +const MAX_SOCKET_BUFFERED_BYTES = 1024 * 1024; +const MAX_ID_LENGTH = 128; +const MAX_VERSION_LENGTH = 128; +const MAX_TRANSCRIPT_LENGTH = 8_192; +const MAX_STATUS_TEXT_LENGTH = 512; +const DEFAULT_SHORTCUT = 'Command+E'; +const MAX_SHORTCUT_LENGTH = 128; +const MAX_APPSHOT_TEXT_LENGTH = 32_000; +const DEFAULT_APPSHOT_TIMEOUT_MS = 15_000; + +function writeLiveHostDiagnostic( + event: string, + details: Readonly>, +): void { + if (process.env['QWEN_LIVE_DIAGNOSTICS'] !== '1') return; + process.stderr.write( + `${JSON.stringify({ + timestamp: new Date().toISOString(), + source: 'live-host-coordinator', + event, + ...details, + })}\n`, + ); +} + +interface LiveCall { + epoch: number; + callId: string; + mode: 'resume' | 'new'; + state: Exclude; + transcript?: string; + caption?: string; + statusText?: string; + coordinator?: LiveSessionLocator; + pendingPermission: boolean; + workers: LiveSessionLocator[]; +} + +interface HostLease { + socket: WebSocket; + hello?: LiveHostHello; + helloTimer: NodeJS.Timeout; + heartbeatTimer?: NodeJS.Timeout; + lastPongAt: number; + pingId?: string; +} + +export interface LiveCallHandlers { + onHostReady?: () => void | Promise; + onStart?: (call: { + epoch: number; + callId: string; + mode: 'resume' | 'new'; + }) => void | Promise; + onStop?: (call: { + epoch: number; + callId: string; + }) => void | { error: string } | Promise; + onInputAudio?: (call: { + epoch: number; + callId: string; + pcm16: Buffer; + }) => boolean; +} + +export interface LiveHostCoordinatorOptions { + daemonInstanceNonce?: string; + getProviderReadiness: () => LiveProviderReadiness; + shortcut?: string; + handlers?: LiveCallHandlers; + helloTimeoutMs?: number; + heartbeatIntervalMs?: number; + heartbeatTimeoutMs?: number; + appshotTimeoutMs?: number; + now?: () => number; +} + +export interface LiveScreenContextCapture { + appName: string; + windowTitle?: string; + accessibilityText: string; + screenshotPath: string; +} + +interface PendingAppshot { + epoch: number; + timer: NodeJS.Timeout; + resolve: (capture: LiveScreenContextCapture) => void; + reject: (error: Error) => void; +} + +interface PendingShortcut { + requestId: string; + shortcut: string; + timer: NodeJS.Timeout; + resolve: (status: LiveStatus) => void; + reject: (error: Error) => void; +} + +export class LiveUnavailableError extends Error { + readonly code = 'live_unavailable' as const; + + constructor(readonly status: LiveStatus) { + super(status.message ?? 'Live Voice is unavailable.'); + this.name = 'LiveUnavailableError'; + } +} + +function isObject(value: unknown): value is Record { + return value !== null && typeof value === 'object' && !Array.isArray(value); +} + +function isBoundedString(value: unknown, maxLength = MAX_ID_LENGTH): boolean { + return ( + typeof value === 'string' && value.length > 0 && value.length <= maxLength + ); +} + +function isPermissionState(value: unknown): value is LivePermissionState { + return ( + value === 'granted' || value === 'denied' || value === 'not_determined' + ); +} + +function parseHello(value: Record): LiveHostHello | undefined { + const permissions = value['permissions']; + const selfChecks = value['selfChecks']; + if ( + value['type'] !== 'host.hello' || + typeof value['protocolVersion'] !== 'number' || + !Number.isInteger(value['protocolVersion']) || + !isBoundedString(value['hostVersion'], MAX_VERSION_LENGTH) || + !isBoundedString(value['bundleId']) || + !isBoundedString(value['instanceNonce']) || + !isObject(permissions) || + !isPermissionState(permissions['microphone']) || + !isPermissionState(permissions['accessibility']) || + !isPermissionState(permissions['screenRecording']) || + !isObject(selfChecks) || + typeof selfChecks['audioInput'] !== 'boolean' || + typeof selfChecks['audioOutput'] !== 'boolean' || + typeof selfChecks['globalShortcut'] !== 'boolean' || + typeof selfChecks['appshot'] !== 'boolean' + ) { + return undefined; + } + return value as unknown as LiveHostHello; +} + +function parseAction( + value: Record, +): LiveHostAction | undefined { + if (value['type'] !== 'host.action') return undefined; + const action = value['action']; + const epoch = value['epoch']; + if ( + epoch !== undefined && + (typeof epoch !== 'number' || !Number.isSafeInteger(epoch) || epoch < 0) + ) { + return undefined; + } + if (action === 'toggle' || action === 'new' || action === 'stop') { + return { + type: 'host.action', + action, + ...(epoch !== undefined ? { epoch } : {}), + }; + } + if (action === 'mute') { + const inputMuted = value['inputMuted']; + const outputMuted = value['outputMuted']; + if ( + (inputMuted === undefined && outputMuted === undefined) || + (inputMuted !== undefined && typeof inputMuted !== 'boolean') || + (outputMuted !== undefined && typeof outputMuted !== 'boolean') + ) { + return undefined; + } + return { + type: 'host.action', + action, + ...(inputMuted !== undefined ? { inputMuted } : {}), + ...(outputMuted !== undefined ? { outputMuted } : {}), + ...(epoch !== undefined ? { epoch } : {}), + }; + } + return undefined; +} + +function parseHostMessage(text: string): LiveHostMessage | undefined { + let value: unknown; + try { + value = JSON.parse(text); + } catch { + return undefined; + } + if (!isObject(value)) return undefined; + if (value['type'] === 'host.hello') return parseHello(value); + if (value['type'] === 'host.action') return parseAction(value); + if (value['type'] === 'host.pong' && isBoundedString(value['pingId'])) { + return { type: 'host.pong', pingId: value['pingId'] as string }; + } + if (value['type'] === 'host.shortcut_result') { + const requestId = value['requestId']; + const shortcut = value['shortcut']; + const success = value['success']; + const error = value['error']; + if ( + isBoundedString(requestId) && + typeof shortcut === 'string' && + shortcut.length <= MAX_SHORTCUT_LENGTH && + typeof success === 'boolean' && + (error === undefined || + (typeof error === 'string' && error.length <= 1_024)) + ) { + return { + type: 'host.shortcut_result', + requestId: requestId as string, + shortcut, + success, + ...(error ? { error } : {}), + }; + } + } + if (value['type'] === 'host.screen_context_result') { + const requestId = value['requestId']; + if (!isBoundedString(requestId)) return undefined; + if (value['success'] === false && isBoundedString(value['error'], 1_024)) { + return { + type: 'host.screen_context_result', + requestId: requestId as string, + success: false, + error: value['error'] as string, + }; + } + if ( + value['success'] === true && + isBoundedString(value['appName'], 512) && + (value['windowTitle'] === undefined || + isBoundedString(value['windowTitle'], 2_048)) && + typeof value['accessibilityText'] === 'string' && + value['accessibilityText'].length <= MAX_APPSHOT_TEXT_LENGTH && + isBoundedString(value['screenshotPath'], 4_096) + ) { + return { + type: 'host.screen_context_result', + requestId: requestId as string, + success: true, + appName: value['appName'] as string, + ...(value['windowTitle'] + ? { windowTitle: value['windowTitle'] as string } + : {}), + accessibilityText: value['accessibilityText'], + screenshotPath: value['screenshotPath'] as string, + }; + } + } + return undefined; +} + +function permissionRequirement( + value: LivePermissionState, +): 'ready' | 'missing' | 'denied' { + if (value === 'granted') return 'ready'; + if (value === 'denied') return 'denied'; + return 'missing'; +} + +function projectStatusForHost(status: LiveStatus): LiveHostStatus { + return { + v: status.v, + available: status.available, + state: status.state, + shortcut: status.shortcut, + ...(status.blocker ? { blocker: status.blocker } : {}), + ...(status.message ? { message: status.message } : {}), + ...(status.callId ? { callId: status.callId } : {}), + ...(status.inputMuted !== undefined + ? { inputMuted: status.inputMuted } + : {}), + ...(status.outputMuted !== undefined + ? { outputMuted: status.outputMuted } + : {}), + ...(status.transcript ? { transcript: status.transcript } : {}), + ...(status.caption ? { caption: status.caption } : {}), + ...(status.statusText ? { statusText: status.statusText } : {}), + ...(status.pendingPermission + ? { pendingPermission: { ...status.pendingPermission } } + : {}), + ...(status.requirements + ? { requirements: { ...status.requirements } } + : {}), + ...(status.host ? { host: { ...status.host } } : {}), + }; +} + +export class LiveHostCoordinator { + readonly daemonInstanceNonce: string; + private readonly now: () => number; + private readonly helloTimeoutMs: number; + private readonly heartbeatIntervalMs: number; + private readonly heartbeatTimeoutMs: number; + private readonly appshotTimeoutMs: number; + private shortcut: string; + private handlers: LiveCallHandlers; + private host?: HostLease; + private hadConnectedHost = false; + private lastHostFailure?: 'host_disconnected' | 'host_version'; + private providerOverride?: LiveProviderReadiness; + private appshotReadiness: LiveAppshotReadiness = { + state: 'unavailable', + message: 'The dedicated Appshot channel has not been verified.', + }; + private call?: LiveCall; + private pendingStartMode?: 'new'; + private deactivating = false; + private nextEpoch = 0; + private inputMuted = false; + private outputMuted = false; + private lastCallError?: string; + private readonly pendingAppshots = new Map(); + private pendingShortcut?: PendingShortcut; + private readonly inactiveWaiters = new Set<() => void>(); + + constructor(private readonly options: LiveHostCoordinatorOptions) { + this.daemonInstanceNonce = options.daemonInstanceNonce ?? randomUUID(); + this.handlers = options.handlers ?? {}; + this.now = options.now ?? Date.now; + this.helloTimeoutMs = options.helloTimeoutMs ?? DEFAULT_HELLO_TIMEOUT_MS; + this.heartbeatIntervalMs = + options.heartbeatIntervalMs ?? DEFAULT_HEARTBEAT_INTERVAL_MS; + this.heartbeatTimeoutMs = + options.heartbeatTimeoutMs ?? DEFAULT_HEARTBEAT_TIMEOUT_MS; + this.appshotTimeoutMs = + options.appshotTimeoutMs ?? DEFAULT_APPSHOT_TIMEOUT_MS; + const shortcut = options.shortcut?.trim(); + this.shortcut = + shortcut && shortcut.length <= MAX_SHORTCUT_LENGTH + ? shortcut + : DEFAULT_SHORTCUT; + } + + setHandlers(handlers: LiveCallHandlers): void { + this.handlers = handlers; + } + + setConfiguredShortcut(shortcut: string): LiveStatus { + const normalized = shortcut.trim(); + if (normalized.length > MAX_SHORTCUT_LENGTH) { + throw new Error('The Live shortcut is too long.'); + } + this.shortcut = normalized; + this.broadcastState(); + return this.getStatus(); + } + + async deactivate(): Promise { + // Host-initiated starts must not re-arm a call while the stop drains: + // state broadcasts during 'stopping' still carry the stopping call's + // epoch, so a host.action would otherwise pass the epoch gate. + this.deactivating = true; + this.pendingStartMode = undefined; + if (this.call) { + const stopped = new Promise((resolve) => { + this.inactiveWaiters.add(resolve); + }); + this.stop(); + await stopped; + } + if (!this.host) return; + const lease = this.host; + if (lease.socket.readyState === WebSocket.OPEN) { + lease.socket.close(1001, 'Live Voice disabled.'); + } + this.detachHost(lease, 'host_disconnected'); + } + + attachHost(socket: WebSocket, daemonNonce: string | undefined): void { + const expectedNonce = Buffer.from(this.daemonInstanceNonce); + const presentedNonce = Buffer.from(daemonNonce ?? ''); + if ( + expectedNonce.byteLength !== presentedNonce.byteLength || + !timingSafeEqual(expectedNonce, presentedNonce) + ) { + socket.close(4003, 'Invalid daemon instance nonce.'); + return; + } + if (this.host && this.isLeaseHealthy(this.host)) { + socket.close(4009, 'A Live Host is already connected.'); + return; + } + if (this.host) this.disconnectHost(this.host, 4008, 'Host lease expired.'); + + // Only an accepted lease revives Live Voice: a fresh host connection is + // the re-activation signal that ends the deactivating window. + this.deactivating = false; + const lease: HostLease = { + socket, + lastPongAt: this.now(), + helloTimer: setTimeout(() => { + if (this.host === lease && !lease.hello) { + socket.close(4000, 'Host hello timeout.'); + this.detachHost(lease, 'host_disconnected'); + } + }, this.helloTimeoutMs), + }; + lease.helloTimer.unref?.(); + this.host = lease; + + socket.on('message', (data, isBinary) => { + if (this.host !== lease) return; + if (isBinary) this.handleAudioFrame(lease, data); + else this.handleTextFrame(lease, data); + }); + socket.on('close', () => { + if (this.host === lease) this.detachHost(lease, 'host_disconnected'); + }); + socket.on('error', () => { + if (this.host === lease) this.detachHost(lease, 'host_disconnected'); + }); + } + + getStatus(): LiveStatus { + return this.buildStatus(true); + } + + private buildStatus(stopOnReadinessLoss: boolean): LiveStatus { + const provider = this.readProviderReadiness(); + const appshot = this.appshotReadiness; + const hello = this.host?.hello; + const requirements: NonNullable = { + host: hello + ? 'ready' + : this.lastHostFailure === 'host_version' + ? 'unavailable' + : this.hadConnectedHost + ? 'unavailable' + : 'missing', + provider: + provider.state === 'ready' + ? 'ready' + : provider.state === 'checking' + ? 'checking' + : 'unavailable', + }; + if (hello) { + requirements.microphone = permissionRequirement( + hello.permissions.microphone, + ); + requirements.accessibility = permissionRequirement( + hello.permissions.accessibility, + ); + requirements.screenRecording = permissionRequirement( + hello.permissions.screenRecording, + ); + requirements.audioInput = hello.selfChecks.audioInput + ? 'ready' + : 'unavailable'; + requirements.audioOutput = hello.selfChecks.audioOutput + ? 'ready' + : 'unavailable'; + requirements.globalShortcut = hello.selfChecks.globalShortcut + ? 'ready' + : 'unavailable'; + requirements.appshot = hello.selfChecks.appshot + ? appshot.state + : 'unavailable'; + } else if (appshot.state !== 'ready') { + requirements.appshot = appshot.state; + } + + const blocker = this.resolveBlocker(provider, appshot, hello); + const providerChecking = provider.state === 'checking'; + const available = !providerChecking && blocker === undefined; + const preserveCheckingCall = + providerChecking && blocker === undefined && this.call !== undefined; + if ( + !available && + this.call && + stopOnReadinessLoss && + !preserveCheckingCall + ) { + this.stopForReadinessLoss(); + } + const active = this.call; + return { + v: 1, + available, + shortcut: this.shortcut, + state: preserveCheckingCall + ? (active?.state ?? 'starting') + : available + ? (active?.state ?? (this.lastCallError ? 'error' : 'idle')) + : 'unavailable', + ...(blocker ? { blocker } : {}), + ...(blocker + ? { message: this.blockerMessage(blocker, provider, appshot, hello) } + : this.lastCallError + ? { message: this.lastCallError } + : {}), + ...(active ? { callId: active.callId } : {}), + inputMuted: this.inputMuted, + outputMuted: this.outputMuted, + ...(active?.transcript ? { transcript: active.transcript } : {}), + ...(active?.caption ? { caption: active.caption } : {}), + ...(active?.statusText ? { statusText: active.statusText } : {}), + ...(active?.pendingPermission && active.coordinator?.workspaceId + ? { + pendingPermission: { + workspaceId: active.coordinator.workspaceId, + sessionId: active.coordinator.sessionId, + }, + } + : {}), + requirements, + ...(hello + ? { + host: { + version: hello.hostVersion, + protocolVersion: hello.protocolVersion, + }, + } + : {}), + }; + } + + start(mode: 'resume' | 'new'): { + epoch: number; + callId: string; + status: LiveStatus; + } { + const status = this.getStatus(); + if (!status.available) throw new LiveUnavailableError(status); + if (mode === 'resume' && this.call) { + return { + epoch: this.call.epoch, + callId: this.call.callId, + status, + }; + } + if (this.call) { + const replacedCall = this.call; + this.pendingStartMode = 'new'; + this.beginCallStop(replacedCall); + const reportedCall = this.call ?? replacedCall; + return { + epoch: reportedCall.epoch, + callId: reportedCall.callId, + status: this.getStatus(), + }; + } + return this.startCall(mode); + } + + private startCall(mode: 'resume' | 'new'): { + epoch: number; + callId: string; + status: LiveStatus; + } { + const call: LiveCall = { + epoch: ++this.nextEpoch, + callId: randomUUID(), + mode, + state: 'starting', + pendingPermission: false, + workers: [], + }; + this.call = call; + this.lastCallError = undefined; + this.broadcastState(); + try { + void Promise.resolve( + this.handlers.onStart?.({ + epoch: call.epoch, + callId: call.callId, + mode, + }), + ).catch(() => { + this.failCall(call.epoch, 'Live Voice failed to start.'); + }); + } catch { + this.failCall(call.epoch, 'Live Voice failed to start.'); + } + return { epoch: call.epoch, callId: call.callId, status: this.getStatus() }; + } + + stop(): LiveStatus { + this.pendingStartMode = undefined; + if (this.call) this.beginCallStop(this.call); + return this.getStatus(); + } + + setMute(update: LiveMuteUpdate): LiveStatus { + if (update.inputMuted !== undefined) { + this.inputMuted = update.inputMuted; + } + if (update.outputMuted !== undefined) { + this.outputMuted = update.outputMuted; + if (update.outputMuted && this.call) this.clearOutput(this.call.epoch); + } + this.broadcastState(); + return this.getStatus(); + } + + setShortcut(shortcut: string): Promise { + const normalized = shortcut.trim(); + if (normalized.length > MAX_SHORTCUT_LENGTH) { + return Promise.reject(new Error('The Live shortcut is too long.')); + } + if (normalized === this.shortcut) { + return Promise.resolve(this.getStatus()); + } + if (this.pendingShortcut) { + return Promise.reject( + new Error('Another Live shortcut update is already in progress.'), + ); + } + const host = this.host; + if (!host?.hello || !this.isLeaseHealthy(host)) { + return Promise.reject( + new Error('Qwen Live Host must be connected to change the shortcut.'), + ); + } + const requestId = randomUUID(); + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { + this.rejectPendingShortcut( + new Error('Qwen Live Host did not confirm the shortcut change.'), + ); + }, DEFAULT_SHORTCUT_TIMEOUT_MS); + timer.unref?.(); + this.pendingShortcut = { + requestId, + shortcut: normalized, + timer, + resolve, + reject, + }; + if ( + !this.sendHost({ + type: 'host.set_shortcut', + requestId, + shortcut: normalized, + }) + ) { + this.rejectPendingShortcut( + new Error('Qwen Live Host is unavailable for shortcut changes.'), + ); + } + }); + } + + setCallState(epoch: number, state: LiveCall['state']): boolean { + if (!this.call || this.call.epoch !== epoch) return false; + if (this.call.state === state) return true; + this.call.state = state; + this.broadcastState(); + return true; + } + + setCoordinator(epoch: number, locator: LiveSessionLocator): boolean { + if (!this.call || this.call.epoch !== epoch) return false; + this.call.coordinator = locator; + this.broadcastState(); + return true; + } + + setPendingPermission(epoch: number, pending: boolean): boolean { + if (!this.call || this.call.epoch !== epoch) return false; + if (this.call.pendingPermission === pending) return true; + this.call.pendingPermission = pending; + this.broadcastState(); + return true; + } + + setTranscript(epoch: number, transcript: string): boolean { + if (!this.call || this.call.epoch !== epoch) return false; + const truncated = transcript.slice(0, MAX_TRANSCRIPT_LENGTH); + if (this.call.transcript === truncated) return true; + this.call.transcript = truncated; + this.broadcastState(); + return true; + } + + setCaption(epoch: number, caption: string): boolean { + if (!this.call || this.call.epoch !== epoch) return false; + const truncated = caption.slice(0, MAX_TRANSCRIPT_LENGTH); + if (this.call.caption === truncated) return true; + this.call.caption = truncated || undefined; + this.broadcastState(); + return true; + } + + setStatusText(epoch: number, statusText?: string): boolean { + if (!this.call || this.call.epoch !== epoch) return false; + const truncated = statusText?.trim().slice(0, MAX_STATUS_TEXT_LENGTH); + if (this.call.statusText === truncated) return true; + this.call.statusText = truncated || undefined; + this.broadcastState(); + return true; + } + + setWorkers(epoch: number, workers: readonly LiveSessionLocator[]): boolean { + if (!this.call || this.call.epoch !== epoch) return false; + this.call.workers = [...workers]; + this.broadcastState(); + return true; + } + + isActiveSession(sessionId: string): boolean { + const call = this.call; + return ( + call !== undefined && + (call.coordinator?.sessionId === sessionId || + call.workers.some((worker) => worker.sessionId === sessionId)) + ); + } + + captureScreenContext( + callerSessionId: string, + ): Promise { + const call = this.call; + const host = this.host; + if ( + !call || + call.coordinator?.sessionId !== callerSessionId || + !host?.hello || + !this.isLeaseHealthy(host) || + host.hello.permissions.accessibility !== 'granted' || + host.hello.permissions.screenRecording !== 'granted' || + !host.hello.selfChecks.appshot + ) { + return Promise.reject( + new Error( + 'Appshot is available only to the active Live session with a ready Host.', + ), + ); + } + const requestId = randomUUID(); + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { + this.pendingAppshots.delete(requestId); + reject(new Error('Live Host Appshot timed out.')); + }, this.appshotTimeoutMs); + timer.unref?.(); + this.pendingAppshots.set(requestId, { + epoch: call.epoch, + timer, + resolve, + reject, + }); + if ( + !this.sendHost({ + type: 'host.capture_screen_context', + requestId, + epoch: call.epoch, + }) + ) { + this.rejectPendingAppshot( + requestId, + new Error('Live Host is unavailable for Appshot.'), + ); + } + }); + } + + setProviderReachability(readiness?: LiveProviderReadiness): void { + this.providerOverride = readiness; + const status = this.getStatus(); + this.sendState(status); + } + + setAppshotReadiness(readiness: LiveAppshotReadiness): void { + this.appshotReadiness = { ...readiness }; + this.sendState(this.getStatus()); + } + + failCall(epoch: number, message = 'Live Voice failed.'): boolean { + if (!this.call || this.call.epoch !== epoch) return false; + const call = this.call; + this.pendingStartMode = undefined; + this.lastCallError = message; + this.beginCallStop(call); + return true; + } + + sendOutputAudio(epoch: number, pcm16: Uint8Array): boolean { + if ( + !this.call || + this.call.epoch !== epoch || + pcm16.byteLength === 0 || + pcm16.byteLength > MAX_DAEMON_AUDIO_BYTES || + pcm16.byteLength % 2 !== 0 + ) { + return false; + } + if (this.outputMuted) return true; + const socket = this.host?.socket; + if ( + !socket || + socket.readyState !== WebSocket.OPEN || + socket.bufferedAmount > MAX_SOCKET_BUFFERED_BYTES + ) { + return false; + } + socket.send(pcm16, { binary: true }); + writeLiveHostDiagnostic('output_audio_sent', { + epoch, + bytes: pcm16.byteLength, + socketBufferedBytes: socket.bufferedAmount, + }); + return true; + } + + clearOutput(epoch: number): void { + if (this.call && this.call.epoch !== epoch) return; + writeLiveHostDiagnostic('clear_output_sent', { epoch }); + this.sendHost({ type: 'host.clear_output', epoch }); + } + + dispose(): void { + this.pendingStartMode = undefined; + if (this.call) this.finishCall(this.call); + if (this.host) { + const lease = this.host; + this.host = undefined; + this.clearLeaseTimers(lease); + if (lease.socket.readyState === WebSocket.OPEN) { + lease.socket.close(1001, 'Daemon shutting down.'); + } + } + this.rejectPendingAppshots(new Error('Live Voice is shutting down.')); + this.rejectPendingShortcut(new Error('Live Voice is shutting down.')); + this.notifyInactive(); + } + + private readProviderReadiness(): LiveProviderReadiness { + if (this.providerOverride) return this.providerOverride; + try { + return this.options.getProviderReadiness(); + } catch { + return { + state: 'unavailable', + blocker: 'provider_config', + message: 'Live provider configuration is invalid.', + }; + } + } + + private resolveBlocker( + provider: LiveProviderReadiness, + appshot: LiveAppshotReadiness, + hello: LiveHostHello | undefined, + ): LiveStatus['blocker'] { + if (provider.state === 'unavailable') { + return provider.blocker ?? 'provider_config'; + } + if (!hello) { + if (this.lastHostFailure === 'host_version') return 'host_version'; + return this.hadConnectedHost ? 'host_disconnected' : 'host_missing'; + } + if (hello.permissions.microphone !== 'granted') { + return 'microphone_permission'; + } + if (hello.permissions.accessibility !== 'granted') { + return 'accessibility_permission'; + } + if (hello.permissions.screenRecording !== 'granted') { + return 'screen_recording_permission'; + } + if (!hello.selfChecks.audioInput) return 'audio_input'; + if (!hello.selfChecks.audioOutput) return 'audio_output'; + if (!hello.selfChecks.globalShortcut) return 'global_shortcut'; + if (!hello.selfChecks.appshot) return 'appshot'; + if (appshot.state !== 'ready') return 'appshot'; + return undefined; + } + + private blockerMessage( + blocker: NonNullable, + provider: LiveProviderReadiness, + appshot: LiveAppshotReadiness, + hello: LiveHostHello | undefined, + ): string { + if ( + (blocker === 'provider_config' || blocker === 'provider_unreachable') && + provider.message + ) { + return provider.message; + } + if (blocker === 'appshot' && hello?.selfChecks.appshot && appshot.message) { + return appshot.message; + } + const messages: Record, string> = { + host_missing: 'Qwen Live Host is not connected.', + host_disconnected: 'Qwen Live Host disconnected.', + host_version: 'Qwen Live Host is not protocol-compatible.', + microphone_permission: 'Microphone permission is required.', + accessibility_permission: 'Accessibility permission is required.', + screen_recording_permission: 'Screen Recording permission is required.', + audio_input: 'Live Host audio input self-check failed.', + audio_output: 'Live Host audio output self-check failed.', + global_shortcut: 'Live Host global shortcut self-check failed.', + appshot: 'Appshot self-check failed.', + provider_config: 'Live provider configuration is invalid.', + provider_unreachable: 'The Live provider is unreachable.', + }; + return messages[blocker]; + } + + private isLeaseHealthy(lease: HostLease): boolean { + return ( + lease.socket.readyState === WebSocket.OPEN && + this.now() - lease.lastPongAt <= this.heartbeatTimeoutMs + ); + } + + private handleTextFrame(lease: HostLease, data: RawData): void { + const text = Buffer.isBuffer(data) + ? data.toString('utf8') + : Array.isArray(data) + ? Buffer.concat(data).toString('utf8') + : Buffer.from(data).toString('utf8'); + if (Buffer.byteLength(text) > MAX_HOST_TEXT_BYTES) { + lease.socket.close(1009, 'Host message is too large.'); + return; + } + const message = parseHostMessage(text); + if (!message) { + this.sendHostError('invalid_message', 'Invalid Live Host message.'); + lease.socket.close(1002, 'Invalid Live Host message.'); + return; + } + if (message.type === 'host.hello') { + this.handleHello(lease, message); + return; + } + if (!lease.hello) { + lease.socket.close(1002, 'host.hello must be the first message.'); + return; + } + if (message.type === 'host.pong') { + if (message.pingId === lease.pingId) { + lease.lastPongAt = this.now(); + lease.pingId = undefined; + } + return; + } + if (message.type === 'host.screen_context_result') { + this.handleScreenContextResult(message); + return; + } + if (message.type === 'host.shortcut_result') { + this.handleShortcutResult(message); + return; + } + this.handleAction(message); + } + + private handleShortcutResult(message: LiveHostShortcutResult): void { + const pending = this.pendingShortcut; + if (!pending || message.requestId !== pending.requestId) return; + this.pendingShortcut = undefined; + clearTimeout(pending.timer); + if (message.shortcut !== pending.shortcut) { + pending.reject( + new Error('Qwen Live Host returned a mismatched shortcut.'), + ); + return; + } + if (!message.success) { + pending.reject( + new Error( + message.error || 'The Live shortcut could not be registered.', + ), + ); + return; + } + this.shortcut = pending.shortcut; + const status = this.getStatus(); + this.sendState(status); + pending.resolve(status); + } + + private handleScreenContextResult( + message: LiveHostScreenContextResult, + ): void { + const pending = this.pendingAppshots.get(message.requestId); + if (!pending) return; + const call = this.call; + if (!call || call.epoch !== pending.epoch) { + this.rejectPendingAppshot( + message.requestId, + new Error('The Live call changed before Appshot completed.'), + ); + return; + } + this.pendingAppshots.delete(message.requestId); + clearTimeout(pending.timer); + if (!message.success) { + pending.reject(new Error(message.error)); + return; + } + pending.resolve({ + appName: message.appName, + ...(message.windowTitle ? { windowTitle: message.windowTitle } : {}), + accessibilityText: message.accessibilityText, + screenshotPath: message.screenshotPath, + }); + } + + private handleHello(lease: HostLease, hello: LiveHostHello): void { + if ( + hello.protocolVersion !== LIVE_HOST_PROTOCOL_VERSION || + hello.bundleId !== LIVE_HOST_BUNDLE_ID || + (lease.hello && lease.hello.instanceNonce !== hello.instanceNonce) + ) { + this.lastHostFailure = 'host_version'; + lease.socket.close(4006, 'Incompatible Live Host.'); + this.detachHost(lease, 'host_version'); + return; + } + lease.hello = hello; + lease.lastPongAt = this.now(); + this.hadConnectedHost = true; + this.lastHostFailure = undefined; + clearTimeout(lease.helloTimer); + if (!lease.heartbeatTimer) { + lease.heartbeatTimer = setInterval( + () => this.heartbeat(lease), + this.heartbeatIntervalMs, + ); + lease.heartbeatTimer.unref?.(); + } + try { + void Promise.resolve(this.handlers.onHostReady?.()).catch( + () => undefined, + ); + } catch { + /* readiness remains fail-closed until a later Host hello */ + } + const status = this.getStatus(); + this.sendHost({ + type: 'host.welcome', + protocolVersion: LIVE_HOST_PROTOCOL_VERSION, + daemonInstanceNonce: this.daemonInstanceNonce, + heartbeatIntervalMs: this.heartbeatIntervalMs, + epoch: this.nextEpoch, + status: projectStatusForHost(status), + }); + this.sendState(status); + } + + private handleAction(action: LiveHostAction): void { + if ( + action.epoch !== undefined && + action.epoch !== (this.call?.epoch ?? this.nextEpoch) + ) { + this.sendHostError('stale_epoch', 'The Live action epoch is stale.'); + return; + } + switch (action.action) { + case 'toggle': + if (this.call) this.stop(); + else this.startFromHost('resume'); + return; + case 'new': + this.startFromHost('new'); + return; + case 'stop': + this.stop(); + return; + case 'mute': + this.setMute(action); + return; + default: + return; + } + } + + private startFromHost(mode: 'resume' | 'new'): void { + if (this.deactivating) return; + try { + this.start(mode); + } catch (error) { + if (error instanceof LiveUnavailableError) { + this.sendState(error.status); + return; + } + this.sendState({ + ...this.getStatus(), + state: 'error', + message: 'Live Voice failed to start.', + }); + } + } + + private handleAudioFrame(lease: HostLease, data: RawData): void { + if (!lease.hello) { + lease.socket.close(1002, 'host.hello must precede audio.'); + return; + } + const audio = Buffer.isBuffer(data) + ? data + : Array.isArray(data) + ? Buffer.concat(data) + : Buffer.from(data); + if ( + audio.byteLength <= LIVE_INPUT_AUDIO_EPOCH_BYTES || + audio.byteLength > MAX_HOST_AUDIO_WIRE_BYTES || + (audio.byteLength - LIVE_INPUT_AUDIO_EPOCH_BYTES) % 2 !== 0 + ) { + lease.socket.close(1009, 'Invalid Live audio frame.'); + return; + } + const encodedEpoch = audio.readBigUInt64BE(0); + if (encodedEpoch > BigInt(Number.MAX_SAFE_INTEGER)) { + lease.socket.close(1009, 'Invalid Live audio frame.'); + return; + } + const epoch = Number(encodedEpoch); + const call = this.call; + if ( + !call || + epoch !== call.epoch || + call.state === 'stopping' || + this.inputMuted + ) { + return; + } + const pcm16 = audio.subarray(LIVE_INPUT_AUDIO_EPOCH_BYTES); + try { + const accepted = this.handlers.onInputAudio?.({ + epoch, + callId: call.callId, + pcm16: Buffer.from(pcm16), + }); + if (accepted === false) { + this.failCall(call.epoch, 'Live Voice audio transport dropped input.'); + } + } catch { + this.failCall(call.epoch, 'Live Voice audio input failed.'); + } + } + + private heartbeat(lease: HostLease): void { + if (this.host !== lease) return; + if (this.now() - lease.lastPongAt > this.heartbeatTimeoutMs) { + this.disconnectHost(lease, 4008, 'Live Host heartbeat timed out.'); + return; + } + const pingId = randomUUID(); + lease.pingId = pingId; + this.sendHost({ type: 'host.ping', pingId }); + } + + private finishCall(call: LiveCall): void { + if (this.call !== call) return; + this.pendingStartMode = undefined; + call.state = 'stopping'; + this.sendState(this.buildStatus(false)); + this.clearOutput(call.epoch); + this.call = undefined; + this.notifyInactive(); + this.rejectPendingAppshots( + new Error('The Live call ended before Appshot completed.'), + call.epoch, + ); + ++this.nextEpoch; + try { + void Promise.resolve( + this.handlers.onStop?.({ epoch: call.epoch, callId: call.callId }), + ).catch(() => {}); + } catch { + // The call is already stopped. Handler failures cannot restore it. + } + this.broadcastState(); + } + + private beginCallStop(call: LiveCall): void { + if (this.call !== call || call.state === 'stopping') return; + call.state = 'stopping'; + this.sendState(this.buildStatus(false)); + this.clearOutput(call.epoch); + let result: void | { error: string } | Promise; + try { + result = this.handlers.onStop?.({ + epoch: call.epoch, + callId: call.callId, + }); + } catch { + this.failStoppingCall(call, 'Live Voice failed to stop safely.'); + return; + } + if (!result || !('then' in result)) { + this.finishStoppingCall(call, result); + return; + } + void Promise.resolve(result).then( + (outcome) => this.finishStoppingCall(call, outcome), + () => this.failStoppingCall(call, 'Live Voice failed to stop safely.'), + ); + } + + private finishStoppingCall( + call: LiveCall, + outcome: void | { error: string }, + ): void { + if (this.call !== call || call.state !== 'stopping') return; + if (outcome?.error) { + this.failStoppingCall(call, outcome.error); + return; + } + this.call = undefined; + this.notifyInactive(); + this.rejectPendingAppshots( + new Error('The Live call ended before Appshot completed.'), + call.epoch, + ); + ++this.nextEpoch; + const pendingStartMode = this.pendingStartMode; + this.pendingStartMode = undefined; + if (pendingStartMode) { + const status = this.getStatus(); + if (status.available) { + this.startCall(pendingStartMode); + return; + } + } + this.broadcastState(); + } + + private failStoppingCall(call: LiveCall, message: string): void { + if (this.call !== call || call.state !== 'stopping') return; + this.pendingStartMode = undefined; + this.call = undefined; + this.notifyInactive(); + this.rejectPendingAppshots(new Error(message), call.epoch); + this.lastCallError = message; + ++this.nextEpoch; + this.broadcastState(); + } + + private stopForReadinessLoss(): void { + const call = this.call; + if (!call) return; + this.pendingStartMode = undefined; + this.beginCallStop(call); + } + + private disconnectHost(lease: HostLease, code: number, reason: string): void { + if (lease.socket.readyState === WebSocket.OPEN) { + lease.socket.close(code, reason); + } + this.detachHost(lease, 'host_disconnected'); + } + + private detachHost( + lease: HostLease, + failure: 'host_disconnected' | 'host_version', + ): void { + if (this.host !== lease) return; + this.host = undefined; + this.clearLeaseTimers(lease); + this.lastHostFailure = failure; + this.rejectPendingAppshots(new Error('Qwen Live Host disconnected.')); + this.rejectPendingShortcut(new Error('Qwen Live Host disconnected.')); + this.stopForReadinessLoss(); + } + + private rejectPendingShortcut(error: Error): void { + const pending = this.pendingShortcut; + if (!pending) return; + this.pendingShortcut = undefined; + clearTimeout(pending.timer); + pending.reject(error); + } + + private rejectPendingAppshot(requestId: string, error: Error): void { + const pending = this.pendingAppshots.get(requestId); + if (!pending) return; + this.pendingAppshots.delete(requestId); + clearTimeout(pending.timer); + pending.reject(error); + } + + private rejectPendingAppshots(error: Error, epoch?: number): void { + for (const [requestId, pending] of this.pendingAppshots) { + if (epoch !== undefined && pending.epoch !== epoch) continue; + this.rejectPendingAppshot(requestId, error); + } + } + + private clearLeaseTimers(lease: HostLease): void { + clearTimeout(lease.helloTimer); + if (lease.heartbeatTimer) clearInterval(lease.heartbeatTimer); + } + + private broadcastState(): void { + this.sendState(this.getStatus()); + } + + private sendState(status: LiveStatus): void { + this.sendHost({ + type: 'host.state', + epoch: this.nextEpoch, + status: projectStatusForHost(status), + }); + } + + private sendHostError( + code: Extract['code'], + message: string, + ): void { + this.sendHost({ type: 'host.error', code, message }); + } + + private sendHost(message: LiveDaemonMessage): boolean { + const lease = this.host; + const socket = lease?.socket; + if (!socket || socket.readyState !== WebSocket.OPEN) return false; + if (socket.bufferedAmount > MAX_SOCKET_BUFFERED_BYTES) { + this.disconnectHost(lease, 4008, 'Live Host is not consuming messages.'); + return false; + } + socket.send(JSON.stringify(message)); + return true; + } + + private notifyInactive(): void { + if (this.call) return; + for (const resolve of this.inactiveWaiters) resolve(); + this.inactiveWaiters.clear(); + } +} diff --git a/packages/qwen-live/src/host/live-host-installer.test.ts b/packages/qwen-live/src/host/live-host-installer.test.ts new file mode 100644 index 00000000000..f00dbc52b6d --- /dev/null +++ b/packages/qwen-live/src/host/live-host-installer.test.ts @@ -0,0 +1,367 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { createHash } from 'node:crypto'; +import * as fsp from 'node:fs/promises'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { describe, expect, it, vi } from 'vitest'; +import { LIVE_HOST_PROTOCOL_VERSION } from './types.js'; +import { + downloadLiveHostRelease, + isExpectedLiveHostSignature, + LiveHostInstaller, + LIVE_HOST_DOWNLOAD_TIMEOUT_MS, + LIVE_HOST_MANIFEST_FETCH_TIMEOUT_MS, + LIVE_HOST_OSS_BASE_URL, + LIVE_HOST_RELEASE_BASE_URL, + parseLiveHostReleaseManifest, + resolveLiveHostAssetUrls, + resolveLiveHostManifestUrls, +} from './live-host-installer.js'; + +const sha = 'a'.repeat(64); + +function manifest() { + return { + schemaVersion: 1, + version: '0.1.0', + protocolVersion: LIVE_HOST_PROTOCOL_VERSION, + bundleId: 'com.alibaba.qwen-code.live-host', + assets: { + arm64: { + name: 'Qwen-Live-Host-arm64.zip', + size: 123, + sha256: sha, + }, + x64: { + name: 'Qwen-Live-Host-x64.zip', + size: 456, + sha256: sha, + }, + }, + }; +} + +function manifestForBytes(version: string, bytes: Buffer) { + const checksum = createHash('sha256').update(bytes).digest('hex'); + return { + ...manifest(), + version, + assets: { + arm64: { + name: 'Qwen-Live-Host-arm64.zip', + size: bytes.byteLength, + sha256: checksum, + }, + x64: { + name: 'Qwen-Live-Host-x64.zip', + size: bytes.byteLength, + sha256: checksum, + }, + }, + }; +} + +describe('LiveHostInstaller', () => { + it('prefers the OSS mirror and retains the independent GitHub fallback', () => { + expect(LIVE_HOST_OSS_BASE_URL).toBe( + 'https://qwen-code-assets.oss-cn-hangzhou.aliyuncs.com/live-host', + ); + expect(LIVE_HOST_RELEASE_BASE_URL).toBe( + 'https://github.com/QwenLM/qwen-code/releases/download/live-host-latest', + ); + expect(resolveLiveHostManifestUrls()).toEqual([ + `${LIVE_HOST_OSS_BASE_URL}/latest/Qwen-Live-Host-manifest.json`, + `${LIVE_HOST_RELEASE_BASE_URL}/Qwen-Live-Host-manifest.json`, + ]); + expect( + resolveLiveHostAssetUrls('0.1.0', 'Qwen-Live-Host-arm64.zip'), + ).toEqual([ + `${LIVE_HOST_OSS_BASE_URL}/v0.1.0/Qwen-Live-Host-arm64.zip`, + `${LIVE_HOST_RELEASE_BASE_URL}/Qwen-Live-Host-arm64.zip`, + ]); + }); + + it('falls back with a matching GitHub manifest and asset pair', async () => { + const ossBytes = Buffer.from('oss-archive'); + const githubBytes = Buffer.from('github-archive'); + const ossManifest = manifestForBytes('0.1.0', ossBytes); + const githubManifest = manifestForBytes('0.2.0', githubBytes); + const fetchImpl = vi + .fn() + .mockResolvedValueOnce(Response.json(ossManifest)) + .mockResolvedValueOnce(new Response(null, { status: 503 })) + .mockResolvedValueOnce(Response.json(githubManifest)) + .mockResolvedValueOnce( + new Response(githubBytes, { + headers: { 'content-length': String(githubBytes.byteLength) }, + }), + ); + const directory = await fsp.mkdtemp( + path.join(os.tmpdir(), 'live-host-download-test-'), + ); + const destination = path.join(directory, 'host.zip'); + + try { + const release = await downloadLiveHostRelease( + 'arm64', + destination, + () => {}, + fetchImpl, + ); + expect(release.manifest).toEqual(githubManifest); + await expect(fsp.readFile(destination)).resolves.toEqual(githubBytes); + expect(fetchImpl.mock.calls.map(([url]) => String(url))).toEqual([ + resolveLiveHostManifestUrls()[0], + resolveLiveHostAssetUrls( + ossManifest.version, + ossManifest.assets.arm64.name, + )[0], + resolveLiveHostManifestUrls()[1], + resolveLiveHostAssetUrls( + githubManifest.version, + githubManifest.assets.arm64.name, + )[1], + ]); + } finally { + await fsp.rm(directory, { recursive: true, force: true }); + } + }); + + it('removes a checksum-invalid OSS archive before falling back', async () => { + const expectedOssBytes = Buffer.from('expected-oss-archive'); + const corruptOssBytes = Buffer.alloc(expectedOssBytes.byteLength, 0x78); + const githubBytes = Buffer.from('github-archive'); + const ossManifest = manifestForBytes('0.1.0', expectedOssBytes); + const githubManifest = manifestForBytes('0.2.0', githubBytes); + const fetchImpl = vi + .fn() + .mockResolvedValueOnce(Response.json(ossManifest)) + .mockResolvedValueOnce( + new Response(corruptOssBytes, { + headers: { 'content-length': String(corruptOssBytes.byteLength) }, + }), + ) + .mockResolvedValueOnce(Response.json(githubManifest)) + .mockResolvedValueOnce( + new Response(githubBytes, { + headers: { 'content-length': String(githubBytes.byteLength) }, + }), + ); + const directory = await fsp.mkdtemp( + path.join(os.tmpdir(), 'live-host-download-test-'), + ); + const destination = path.join(directory, 'host.zip'); + + try { + const release = await downloadLiveHostRelease( + 'arm64', + destination, + () => {}, + fetchImpl, + ); + expect(release.manifest).toEqual(githubManifest); + await expect(fsp.readFile(destination)).resolves.toEqual(githubBytes); + } finally { + await fsp.rm(directory, { recursive: true, force: true }); + } + }); + + it('reports both source failures and removes the destination', async () => { + const fetchImpl = vi + .fn() + .mockResolvedValue(new Response(null, { status: 503 })); + const directory = await fsp.mkdtemp( + path.join(os.tmpdir(), 'live-host-download-test-'), + ); + const destination = path.join(directory, 'host.zip'); + + try { + await expect( + downloadLiveHostRelease('arm64', destination, () => {}, fetchImpl), + ).rejects.toMatchObject({ + message: + 'Qwen Live Host download failed. OSS: Live Host manifest download failed (503). GitHub: Live Host manifest download failed (503).', + errors: [ + { message: 'OSS: Live Host manifest download failed (503).' }, + { message: 'GitHub: Live Host manifest download failed (503).' }, + ], + }); + await expect(fsp.stat(destination)).rejects.toMatchObject({ + code: 'ENOENT', + }); + } finally { + await fsp.rm(directory, { recursive: true, force: true }); + } + }); + + it('passes separate bounded signals to manifest and archive requests', async () => { + const bytes = Buffer.from('signed-live-host-archive'); + const expected = manifestForBytes('0.1.0', bytes); + const manifestSignal = AbortSignal.abort('manifest'); + const downloadSignal = AbortSignal.abort('download'); + const timeout = vi + .spyOn(AbortSignal, 'timeout') + .mockReturnValueOnce(manifestSignal) + .mockReturnValueOnce(downloadSignal); + const fetchImpl = vi + .fn() + .mockResolvedValueOnce(Response.json(expected)) + .mockResolvedValueOnce(new Response(bytes)); + const directory = await fsp.mkdtemp( + path.join(os.tmpdir(), 'live-host-download-test-'), + ); + const destination = path.join(directory, 'host.zip'); + + try { + await downloadLiveHostRelease('arm64', destination, () => {}, fetchImpl); + expect(LIVE_HOST_MANIFEST_FETCH_TIMEOUT_MS).toBe(5 * 60 * 1000); + expect(LIVE_HOST_DOWNLOAD_TIMEOUT_MS).toBe(60 * 60 * 1000); + expect(fetchImpl.mock.calls[0]?.[1]?.signal).toBe(manifestSignal); + expect(fetchImpl.mock.calls[1]?.[1]?.signal).toBe(downloadSignal); + } finally { + timeout.mockRestore(); + await fsp.rm(directory, { recursive: true, force: true }); + } + }); + + it('accepts only the Qwen Developer ID team', () => { + expect( + isExpectedLiveHostSignature( + [ + 'Authority=Developer ID Application: Alibaba Cloud (Singapore) Private Limited (NF4574S59H)', + 'TeamIdentifier=NF4574S59H', + ].join('\n'), + ), + ).toBe(true); + expect( + isExpectedLiveHostSignature( + 'Authority=Developer ID Application: Other\nTeamIdentifier=OTHER12345', + ), + ).toBe(false); + expect( + isExpectedLiveHostSignature('Signature=adhoc\nTeamIdentifier=NF4574S59H'), + ).toBe(false); + }); + + it('accepts only the fixed compatible release manifest', () => { + expect(parseLiveHostReleaseManifest(manifest())).toEqual(manifest()); + expect(() => + parseLiveHostReleaseManifest({ + ...manifest(), + protocolVersion: LIVE_HOST_PROTOCOL_VERSION + 1, + }), + ).toThrow(/incompatible/); + expect(() => + parseLiveHostReleaseManifest({ + ...manifest(), + assets: { + ...manifest().assets, + arm64: { ...manifest().assets.arm64, name: 'other.zip' }, + }, + }), + ).toThrow(/asset/); + }); + + it('launches an existing verified installation without downloading', async () => { + const inspectInstalled = vi.fn(async () => ({ version: '0.1.0' })); + const installLatest = vi.fn(); + const launch = vi.fn(async () => {}); + const installer = new LiveHostInstaller({ + platform: 'darwin', + architecture: 'arm64', + inspectInstalled, + installLatest, + launch, + }); + + await expect(installer.ensureInstalled()).resolves.toEqual({ + state: 'installed', + version: '0.1.0', + }); + expect(installLatest).not.toHaveBeenCalled(); + expect(launch).toHaveBeenCalledOnce(); + }); + + it('coalesces concurrent installs and exposes progress', async () => { + let finish: ((value: { version: string }) => void) | undefined; + const installLatest = vi.fn( + async ( + _architecture: 'arm64' | 'x64', + onStatus: (status: { state: 'downloading'; progress: number }) => void, + ) => { + onStatus({ state: 'downloading', progress: 0.5 }); + return await new Promise<{ version: string }>((resolve) => { + finish = resolve; + }); + }, + ); + const installer = new LiveHostInstaller({ + platform: 'darwin', + architecture: 'x64', + inspectInstalled: async () => undefined, + installLatest, + launch: async () => {}, + }); + + const first = installer.ensureInstalled(); + const second = installer.ensureInstalled(); + await vi.waitFor(() => { + expect(installer.getStatus()).toEqual({ + state: 'downloading', + progress: 0.5, + }); + }); + finish?.({ version: '0.1.0' }); + await expect(first).resolves.toMatchObject({ state: 'installed' }); + await expect(second).resolves.toMatchObject({ state: 'installed' }); + expect(installLatest).toHaveBeenCalledOnce(); + }); + + it('fails closed on unsupported platforms and architectures', async () => { + const installLatest = vi.fn(); + const linux = new LiveHostInstaller({ + platform: 'linux', + architecture: 'x64', + installLatest, + }); + await expect(linux.ensureInstalled()).resolves.toMatchObject({ + state: 'error', + retryable: false, + }); + + const unsupported = new LiveHostInstaller({ + platform: 'darwin', + architecture: 'ia32', + inspectInstalled: async () => undefined, + installLatest, + }); + await expect(unsupported.ensureInstalled()).resolves.toMatchObject({ + state: 'error', + retryable: true, + }); + expect(installLatest).not.toHaveBeenCalled(); + }); + + it('keeps an installation failure retryable', async () => { + const installer = new LiveHostInstaller({ + platform: 'darwin', + architecture: 'arm64', + inspectInstalled: async () => undefined, + installLatest: async () => { + throw new Error('checksum verification failed'); + }, + launch: async () => {}, + }); + + await expect(installer.ensureInstalled()).resolves.toEqual({ + state: 'error', + message: 'checksum verification failed', + retryable: true, + }); + }); +}); diff --git a/packages/qwen-live/src/host/live-host-installer.ts b/packages/qwen-live/src/host/live-host-installer.ts new file mode 100644 index 00000000000..02214910848 --- /dev/null +++ b/packages/qwen-live/src/host/live-host-installer.ts @@ -0,0 +1,504 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { execFile } from 'node:child_process'; +import { createHash, randomUUID } from 'node:crypto'; +import * as fs from 'node:fs'; +import * as fsp from 'node:fs/promises'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { Readable, Transform } from 'node:stream'; +import { pipeline } from 'node:stream/promises'; +import { promisify } from 'node:util'; +import { LIVE_HOST_PROTOCOL_VERSION } from './types.js'; + +const execFileAsync = promisify(execFile); + +export const LIVE_HOST_BUNDLE_ID = 'com.alibaba.qwen-code.live-host'; +export const LIVE_HOST_TEAM_IDENTIFIER = 'NF4574S59H'; +export const LIVE_HOST_APP_PATH = '/Applications/Qwen Live Host.app'; +export const LIVE_HOST_OSS_BASE_URL = + 'https://qwen-code-assets.oss-cn-hangzhou.aliyuncs.com/live-host'; +export const LIVE_HOST_RELEASE_BASE_URL = + 'https://github.com/QwenLM/qwen-code/releases/download/live-host-latest'; +export const LIVE_HOST_MANIFEST_NAME = 'Qwen-Live-Host-manifest.json'; +export const LIVE_HOST_MANIFEST_FETCH_TIMEOUT_MS = 5 * 60 * 1000; +export const LIVE_HOST_DOWNLOAD_TIMEOUT_MS = 60 * 60 * 1000; + +const LIVE_HOST_APP_NAME = 'Qwen Live Host.app'; +const MAX_DOWNLOAD_BYTES = 512 * 1024 * 1024; +const COMMAND_TIMEOUT_MS = 2 * 60 * 1000; +const SHA256_PATTERN = /^[a-f0-9]{64}$/; +const VERSION_PATTERN = /^\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?$/; + +export type LiveHostArchitecture = 'arm64' | 'x64'; + +export type LiveHostInstallState = + | 'missing' + | 'checking' + | 'downloading' + | 'verifying' + | 'installing' + | 'launching' + | 'installed' + | 'error'; + +export interface LiveHostInstallStatus { + state: LiveHostInstallState; + version?: string; + progress?: number; + message?: string; + retryable?: boolean; +} + +export interface LiveHostReleaseAsset { + name: string; + size: number; + sha256: string; +} + +export interface LiveHostReleaseManifest { + schemaVersion: 1; + version: string; + protocolVersion: number; + bundleId: string; + assets: Record; +} + +interface InstalledLiveHost { + version: string; +} + +export interface LiveHostInstallerDeps { + platform?: NodeJS.Platform; + architecture?: string; + inspectInstalled?: () => Promise; + installLatest?: ( + architecture: LiveHostArchitecture, + onStatus: (status: LiveHostInstallStatus) => void, + ) => Promise; + launch?: () => Promise; +} + +export function isExpectedLiveHostSignature(output: string): boolean { + return ( + /^Authority=Developer ID Application:/m.test(output) && + new RegExp(`^TeamIdentifier=${LIVE_HOST_TEAM_IDENTIFIER}$`, 'm').test( + output, + ) + ); +} + +function architecture(value: string): LiveHostArchitecture { + if (value === 'arm64' || value === 'x64') return value; + throw new Error(`Qwen Live Host is unavailable for architecture ${value}.`); +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function parseAsset( + value: unknown, + expectedName: string, +): LiveHostReleaseAsset { + if (!isRecord(value)) throw new Error('Live Host manifest asset is invalid.'); + const name = value['name']; + const size = value['size']; + const sha256 = value['sha256']; + if ( + name !== expectedName || + !Number.isSafeInteger(size) || + Number(size) <= 0 || + Number(size) > MAX_DOWNLOAD_BYTES || + typeof sha256 !== 'string' || + !SHA256_PATTERN.test(sha256) + ) { + throw new Error('Live Host manifest asset is invalid.'); + } + return { name, size: Number(size), sha256 }; +} + +export function parseLiveHostReleaseManifest( + value: unknown, +): LiveHostReleaseManifest { + if (!isRecord(value) || !isRecord(value['assets'])) { + throw new Error('Live Host manifest is invalid.'); + } + const version = value['version']; + if ( + value['schemaVersion'] !== 1 || + typeof version !== 'string' || + !VERSION_PATTERN.test(version) || + value['protocolVersion'] !== LIVE_HOST_PROTOCOL_VERSION || + value['bundleId'] !== LIVE_HOST_BUNDLE_ID + ) { + throw new Error('Live Host manifest is incompatible.'); + } + return { + schemaVersion: 1, + version, + protocolVersion: LIVE_HOST_PROTOCOL_VERSION, + bundleId: LIVE_HOST_BUNDLE_ID, + assets: { + arm64: parseAsset(value['assets']['arm64'], 'Qwen-Live-Host-arm64.zip'), + x64: parseAsset(value['assets']['x64'], 'Qwen-Live-Host-x64.zip'), + }, + }; +} + +async function run(file: string, args: string[]): Promise { + const { stdout } = await execFileAsync(file, args, { + encoding: 'utf8', + timeout: COMMAND_TIMEOUT_MS, + maxBuffer: 1024 * 1024, + }); + return stdout.trim(); +} + +async function readBundleValue(appPath: string, key: string): Promise { + return await run('/usr/bin/plutil', [ + '-extract', + key, + 'raw', + '-o', + '-', + path.join(appPath, 'Contents', 'Info.plist'), + ]); +} + +async function inspectApp(appPath: string): Promise { + const stat = await fsp.lstat(appPath); + if (!stat.isDirectory() || stat.isSymbolicLink()) { + throw new Error('Qwen Live Host installation is not a regular app bundle.'); + } + const bundleId = await readBundleValue(appPath, 'CFBundleIdentifier'); + if (bundleId !== LIVE_HOST_BUNDLE_ID) { + throw new Error('Qwen Live Host bundle identity is invalid.'); + } + const version = await readBundleValue(appPath, 'CFBundleShortVersionString'); + if (!VERSION_PATTERN.test(version)) { + throw new Error('Qwen Live Host version is invalid.'); + } + await run('/usr/bin/codesign', [ + '--verify', + '--deep', + '--strict', + '--verbose=2', + appPath, + ]); + const signature = await execFileAsync( + '/usr/bin/codesign', + ['-dv', '--verbose=4', appPath], + { + encoding: 'utf8', + timeout: COMMAND_TIMEOUT_MS, + maxBuffer: 1024 * 1024, + }, + ); + const signatureOutput = `${signature.stdout}${signature.stderr}`; + if (!isExpectedLiveHostSignature(signatureOutput)) { + throw new Error('Qwen Live Host signing identity is invalid.'); + } + await run('/usr/sbin/spctl', ['-a', '-t', 'exec', appPath]); + return { version }; +} + +async function inspectInstalledHost(): Promise { + try { + return await inspectApp(LIVE_HOST_APP_PATH); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return undefined; + throw error; + } +} + +export function resolveLiveHostManifestUrls(): [string, string] { + return [ + `${LIVE_HOST_OSS_BASE_URL}/latest/${LIVE_HOST_MANIFEST_NAME}`, + `${LIVE_HOST_RELEASE_BASE_URL}/${LIVE_HOST_MANIFEST_NAME}`, + ]; +} + +export function resolveLiveHostAssetUrls( + version: string, + assetName: string, +): [string, string] { + return [ + `${LIVE_HOST_OSS_BASE_URL}/v${version}/${assetName}`, + `${LIVE_HOST_RELEASE_BASE_URL}/${assetName}`, + ]; +} + +async function fetchManifest( + url: string, + fetchImpl: typeof fetch, +): Promise { + const response = await fetchImpl(url, { + signal: AbortSignal.timeout(LIVE_HOST_MANIFEST_FETCH_TIMEOUT_MS), + }); + if (!response.ok) { + await response.body?.cancel().catch(() => {}); + throw new Error(`Live Host manifest download failed (${response.status}).`); + } + return parseLiveHostReleaseManifest(await response.json()); +} + +async function downloadAsset( + asset: LiveHostReleaseAsset, + url: string, + destination: string, + onProgress: (progress: number) => void, + fetchImpl: typeof fetch, +): Promise { + const response = await fetchImpl(url, { + signal: AbortSignal.timeout(LIVE_HOST_DOWNLOAD_TIMEOUT_MS), + }); + if (!response.ok || !response.body) { + await response.body?.cancel().catch(() => {}); + throw new Error(`Live Host download failed (${response.status}).`); + } + const contentLength = Number(response.headers.get('content-length')); + if ( + Number.isFinite(contentLength) && + contentLength > 0 && + contentLength !== asset.size + ) { + await response.body.cancel().catch(() => {}); + throw new Error('Live Host download size does not match its manifest.'); + } + const hash = createHash('sha256'); + let received = 0; + const guard = new Transform({ + transform(chunk: Buffer, _encoding, callback) { + received += chunk.byteLength; + if (received > asset.size || received > MAX_DOWNLOAD_BYTES) { + callback(new Error('Live Host download exceeded its manifest size.')); + return; + } + hash.update(chunk); + onProgress(Math.min(1, received / asset.size)); + callback(null, chunk); + }, + }); + await pipeline( + Readable.fromWeb(response.body as Parameters[0]), + guard, + fs.createWriteStream(destination, { flags: 'wx', mode: 0o600 }), + ); + if (received !== asset.size || hash.digest('hex') !== asset.sha256) { + throw new Error('Live Host checksum verification failed.'); + } +} + +export async function downloadLiveHostRelease( + currentArchitecture: LiveHostArchitecture, + destination: string, + onProgress: (progress: number) => void, + fetchImpl: typeof fetch = fetch, +): Promise<{ + manifest: LiveHostReleaseManifest; + asset: LiveHostReleaseAsset; +}> { + const manifestUrls = resolveLiveHostManifestUrls(); + const labels = ['OSS', 'GitHub']; + const errors: Error[] = []; + + for (let index = 0; index < manifestUrls.length; index += 1) { + try { + await fsp.rm(destination, { force: true }); + const manifest = await fetchManifest(manifestUrls[index], fetchImpl); + const asset = manifest.assets[currentArchitecture]; + const assetUrl = resolveLiveHostAssetUrls(manifest.version, asset.name)[ + index + ]; + await downloadAsset(asset, assetUrl, destination, onProgress, fetchImpl); + return { manifest, asset }; + } catch (error) { + errors.push( + new Error(`${labels[index]}: ${errorMessage(error)}`, { cause: error }), + ); + } + } + + await fsp.rm(destination, { force: true }); + throw new AggregateError( + errors, + `Qwen Live Host download failed. ${errors + .map((error) => error.message) + .join(' ')}`, + ); +} + +async function installLatestHost( + currentArchitecture: LiveHostArchitecture, + onStatus: (status: LiveHostInstallStatus) => void, +): Promise { + const temporaryDirectory = await fsp.mkdtemp( + path.join(os.tmpdir(), 'qwen-live-host-'), + ); + const archivePath = path.join(temporaryDirectory, 'Qwen-Live-Host.zip'); + const extractedPath = path.join(temporaryDirectory, 'extracted'); + const candidatePath = path.join(extractedPath, LIVE_HOST_APP_NAME); + const stagingPath = `/Applications/.Qwen Live Host.install-${randomUUID()}.app`; + const backupPath = `/Applications/.Qwen Live Host.backup-${randomUUID()}.app`; + let movedExisting = false; + let installedCandidate = false; + try { + onStatus({ state: 'downloading', progress: 0 }); + const { manifest } = await downloadLiveHostRelease( + currentArchitecture, + archivePath, + (progress) => { + onStatus({ state: 'downloading', progress }); + }, + ); + onStatus({ state: 'verifying' }); + await fsp.mkdir(extractedPath, { mode: 0o700 }); + await run('/usr/bin/ditto', ['-x', '-k', archivePath, extractedPath]); + const candidate = await inspectApp(candidatePath); + if (candidate.version !== manifest.version) { + throw new Error('Live Host package version does not match its manifest.'); + } + onStatus({ state: 'installing', version: manifest.version }); + await run('/usr/bin/ditto', [candidatePath, stagingPath]); + await inspectApp(stagingPath); + try { + await fsp.rename(LIVE_HOST_APP_PATH, backupPath); + movedExisting = true; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + } + await fsp.rename(stagingPath, LIVE_HOST_APP_PATH); + installedCandidate = true; + const installed = await inspectApp(LIVE_HOST_APP_PATH); + if (installed.version !== manifest.version) { + throw new Error('Installed Live Host version is invalid.'); + } + if (movedExisting) { + await fsp.rm(backupPath, { recursive: true, force: true }); + } + return installed; + } catch (error) { + if (installedCandidate) { + await fsp.rm(LIVE_HOST_APP_PATH, { recursive: true, force: true }); + } + if (movedExisting) { + await fsp.rename(backupPath, LIVE_HOST_APP_PATH).catch(() => {}); + } + throw error; + } finally { + await fsp.rm(stagingPath, { recursive: true, force: true }).catch(() => {}); + await fsp.rm(temporaryDirectory, { recursive: true, force: true }); + } +} + +async function launchInstalledHost(): Promise { + await run('/usr/bin/open', [LIVE_HOST_APP_PATH]); +} + +function errorMessage(error: unknown): string { + if (error instanceof Error && error.message) return error.message; + return 'Qwen Live Host setup failed.'; +} + +export class LiveHostInstaller { + private status: LiveHostInstallStatus = { state: 'missing' }; + private operation: Promise | undefined; + private readonly platform: NodeJS.Platform; + private readonly currentArchitecture: string; + private readonly inspectInstalled: () => Promise< + InstalledLiveHost | undefined + >; + private readonly installLatest: NonNullable< + LiveHostInstallerDeps['installLatest'] + >; + private readonly launchHost: () => Promise; + + constructor(deps: LiveHostInstallerDeps = {}) { + this.platform = deps.platform ?? process.platform; + this.currentArchitecture = deps.architecture ?? process.arch; + this.inspectInstalled = deps.inspectInstalled ?? inspectInstalledHost; + this.installLatest = deps.installLatest ?? installLatestHost; + this.launchHost = deps.launch ?? launchInstalledHost; + } + + getStatus(): LiveHostInstallStatus { + return { ...this.status }; + } + + async refresh(): Promise { + if (this.operation) return await this.operation; + if (this.platform !== 'darwin') { + return this.setError('Qwen Live Host is available only on macOS.', false); + } + this.status = { state: 'checking' }; + try { + const installed = await this.inspectInstalled(); + this.status = installed + ? { state: 'installed', version: installed.version } + : { state: 'missing' }; + } catch (error) { + this.setError(errorMessage(error), true); + } + return this.getStatus(); + } + + ensureInstalled(force = false): Promise { + if (this.operation) return this.operation; + const operation = this.runInstall(force).finally(() => { + if (this.operation === operation) this.operation = undefined; + }); + this.operation = operation; + return operation; + } + + async launch(): Promise { + if (this.platform !== 'darwin') { + return this.setError('Qwen Live Host is available only on macOS.', false); + } + if (this.operation) return await this.operation; + try { + const installed = await this.inspectInstalled(); + if (!installed) + return this.setError('Qwen Live Host is not installed.', true); + this.status = { state: 'launching', version: installed.version }; + await this.launchHost(); + this.status = { state: 'installed', version: installed.version }; + } catch (error) { + this.setError(errorMessage(error), true); + } + return this.getStatus(); + } + + private async runInstall(force: boolean): Promise { + if (this.platform !== 'darwin') { + return this.setError('Qwen Live Host is available only on macOS.', false); + } + let currentArchitecture: LiveHostArchitecture; + try { + currentArchitecture = architecture(this.currentArchitecture); + this.status = { state: 'checking' }; + const installed = force ? undefined : await this.inspectInstalled(); + const ready = + installed ?? + (await this.installLatest(currentArchitecture, (status) => { + this.status = { ...status }; + })); + this.status = { state: 'launching', version: ready.version }; + await this.launchHost(); + this.status = { state: 'installed', version: ready.version }; + } catch (error) { + this.setError(errorMessage(error), true); + } + return this.getStatus(); + } + + private setError(message: string, retryable: boolean): LiveHostInstallStatus { + this.status = { state: 'error', message, retryable }; + return this.getStatus(); + } +} diff --git a/packages/qwen-live/src/host/types.ts b/packages/qwen-live/src/host/types.ts new file mode 100644 index 00000000000..0d01e7228a4 --- /dev/null +++ b/packages/qwen-live/src/host/types.ts @@ -0,0 +1,197 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +export const LIVE_HOST_PROTOCOL_VERSION = 6 as const; +export const LIVE_HOST_BUNDLE_ID = 'com.alibaba.qwen-code.live-host' as const; +export const LIVE_INPUT_AUDIO_EPOCH_BYTES = 8; + +export type LiveState = + | 'unavailable' + | 'idle' + | 'starting' + | 'listening' + | 'thinking' + | 'speaking' + | 'stopping' + | 'error'; + +export type LiveBlocker = + | 'host_missing' + | 'host_disconnected' + | 'host_version' + | 'microphone_permission' + | 'accessibility_permission' + | 'screen_recording_permission' + | 'audio_input' + | 'audio_output' + | 'global_shortcut' + | 'appshot' + | 'provider_config' + | 'provider_unreachable'; + +export type LiveRequirementState = + | 'ready' + | 'missing' + | 'denied' + | 'unavailable' + | 'checking'; + +export interface LiveSessionLocator { + workspaceCwd: string; + workspaceId?: string; + sessionId: string; +} + +export interface LiveStatus { + v: 1; + available: boolean; + state: LiveState; + shortcut: string; + blocker?: LiveBlocker; + message?: string; + callId?: string; + inputMuted?: boolean; + outputMuted?: boolean; + transcript?: string; + caption?: string; + statusText?: string; + pendingPermission?: { + workspaceId: string; + sessionId: string; + }; + requirements?: Partial< + Record< + | 'host' + | 'microphone' + | 'accessibility' + | 'screenRecording' + | 'audioInput' + | 'audioOutput' + | 'globalShortcut' + | 'appshot' + | 'provider', + LiveRequirementState + > + >; + host?: { + version?: string; + protocolVersion?: number; + }; +} + +export type LiveHostStatus = LiveStatus; + +export type LivePermissionState = 'granted' | 'denied' | 'not_determined'; + +export interface LiveHostHello { + type: 'host.hello'; + protocolVersion: number; + hostVersion: string; + bundleId: string; + instanceNonce: string; + permissions: { + microphone: LivePermissionState; + accessibility: LivePermissionState; + screenRecording: LivePermissionState; + }; + selfChecks: { + audioInput: boolean; + audioOutput: boolean; + globalShortcut: boolean; + appshot: boolean; + }; +} + +export type LiveHostAction = + | { + type: 'host.action'; + action: 'toggle' | 'new' | 'stop'; + epoch?: number; + } + | { + type: 'host.action'; + action: 'mute'; + inputMuted?: boolean; + outputMuted?: boolean; + epoch?: number; + }; + +export interface LiveHostPong { + type: 'host.pong'; + pingId: string; +} + +export interface LiveHostShortcutResult { + type: 'host.shortcut_result'; + requestId: string; + shortcut: string; + success: boolean; + error?: string; +} + +export type LiveHostScreenContextResult = + | { + type: 'host.screen_context_result'; + requestId: string; + success: true; + appName: string; + windowTitle?: string; + accessibilityText: string; + screenshotPath: string; + } + | { + type: 'host.screen_context_result'; + requestId: string; + success: false; + error: string; + }; + +export type LiveHostMessage = + | LiveHostHello + | LiveHostAction + | LiveHostPong + | LiveHostShortcutResult + | LiveHostScreenContextResult; + +export type LiveDaemonMessage = + | { + type: 'host.welcome'; + protocolVersion: typeof LIVE_HOST_PROTOCOL_VERSION; + daemonInstanceNonce: string; + heartbeatIntervalMs: number; + epoch: number; + status: LiveHostStatus; + } + | { type: 'host.state'; epoch: number; status: LiveHostStatus } + | { type: 'host.ping'; pingId: string } + | { type: 'host.clear_output'; epoch: number } + | { type: 'host.set_shortcut'; requestId: string; shortcut: string } + | { + type: 'host.capture_screen_context'; + requestId: string; + epoch: number; + } + | { + type: 'host.error'; + code: 'invalid_message' | 'stale_epoch'; + message: string; + }; + +export interface LiveMuteUpdate { + inputMuted?: boolean; + outputMuted?: boolean; +} + +export interface LiveProviderReadiness { + state: 'ready' | 'checking' | 'unavailable'; + blocker?: Extract; + message?: string; +} + +export interface LiveAppshotReadiness { + state: 'ready' | 'checking' | 'unavailable'; + message?: string; +} diff --git a/packages/qwen-live/src/index.ts b/packages/qwen-live/src/index.ts new file mode 100644 index 00000000000..9b5ff719056 --- /dev/null +++ b/packages/qwen-live/src/index.ts @@ -0,0 +1,112 @@ +#!/usr/bin/env node +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * `qwen-live` bin entry: load config, start the daemon, exit cleanly on + * SIGINT/SIGTERM. + */ + +import { realpathSync, statSync } from 'node:fs'; +import { join } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { loadConfig } from './config.js'; +import { LiveDaemon } from './daemon.js'; +import { LiveLogger } from './logger.js'; + +export { loadConfig, type LiveConfig } from './config.js'; +export { LiveDaemon } from './daemon.js'; +export type { + BackendAdaptor, + BackendCapabilities, + BackendEvent, + BackendHandle, +} from './adaptor/types.js'; + +async function main(): Promise { + const logger = new LiveLogger(); + // A stray rejection in a background chain (event pump, auto-approval) + // must be diagnosable, not process-fatal. + process.on('unhandledRejection', (reason) => { + logger.error( + `unhandled rejection: ${ + reason instanceof Error + ? (reason.stack ?? reason.message) + : String(reason) + }`, + ); + }); + let daemon: LiveDaemon; + try { + daemon = new LiveDaemon(loadConfig()); + } catch (error) { + logger.error(error instanceof Error ? error.message : String(error)); + process.exitCode = 1; + return; + } + + let shuttingDown = false; + const shutdown = (signal: string) => { + if (shuttingDown) return; + shuttingDown = true; + logger.info(`received ${signal}, shutting down`); + daemon + .stop() + .catch((error: unknown) => { + logger.error( + `shutdown failed: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + }) + .finally(() => { + process.exit(0); + }); + }; + process.on('SIGINT', () => { + shutdown('SIGINT'); + }); + process.on('SIGTERM', () => { + shutdown('SIGTERM'); + }); + + try { + await daemon.start(); + } catch (error) { + logger.error(error instanceof Error ? error.message : String(error)); + await daemon.stop().catch(() => undefined); + process.exitCode = 1; + } +} + +// Only run as a daemon when invoked as the bin, not when imported. npm +// installs bins as symlinks and Node resolves import.meta.url through them, +// so compare against the realpath (same pattern as packages/cli/src/cli.ts); +// pathToFileURL also percent-encodes metacharacters (# ? %) the way Node did +// when it formed import.meta.url. +let invokedDirectly = false; +if (process.argv[1] !== undefined) { + try { + const entry = realpathSync(process.argv[1]); + // Direct file invocation… + invokedDirectly = import.meta.url === pathToFileURL(entry).href; + // …or directory-form invocation (`node packages/qwen-live`): Node + // resolves the directory against package.json main; compare against + // the built entry the bin ships so main() still runs. + if ( + !invokedDirectly && + statSync(entry, { throwIfNoEntry: false })?.isDirectory() + ) { + invokedDirectly = + import.meta.url === pathToFileURL(join(entry, 'dist', 'index.js')).href; + } + } catch { + invokedDirectly = import.meta.url === pathToFileURL(process.argv[1]).href; + } +} +if (invokedDirectly) { + void main(); +} diff --git a/packages/qwen-live/src/log/session-log.test.ts b/packages/qwen-live/src/log/session-log.test.ts new file mode 100644 index 00000000000..af8aea03f2b --- /dev/null +++ b/packages/qwen-live/src/log/session-log.test.ts @@ -0,0 +1,163 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { mkdtemp, readFile, rm, stat } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { SessionLog } from './session-log.js'; + +interface LoggedLine { + ts: number; + seq: number; + type: string; + payload: Record; +} + +const temporaryDirectories: string[] = []; + +async function temporaryDirectory(): Promise { + const directory = await mkdtemp(join(tmpdir(), 'qwen-live-log-')); + temporaryDirectories.push(directory); + return directory; +} + +async function readLines(path: string): Promise { + const raw = await readFile(path, 'utf8'); + return raw + .split('\n') + .filter((line) => line.length > 0) + .map((line) => JSON.parse(line) as LoggedLine); +} + +afterEach(async () => { + await Promise.all( + temporaryDirectories + .splice(0) + .map((directory) => rm(directory, { recursive: true, force: true })), + ); +}); + +describe('SessionLog', () => { + it('writes one JSON line per event with ts, monotonic seq, type, and payload', async () => { + const base = await temporaryDirectory(); + let tick = 100; + const log = new SessionLog({ + directory: join(base, 'logs'), + liveSessionId: 'live_test', + now: () => tick, + }); + + log.write('session.start', { liveSessionId: 'live_test' }); + tick = 250; + log.write('tool.call', { name: 'start_job' }); + tick = 400; + log.write('session.end', {}); + await log.close(); + + expect(log.filePath).toBe(join(base, 'logs', 'live_test.jsonl')); + const lines = await readLines(log.filePath); + expect(lines).toEqual([ + { + ts: 100, + seq: 1, + type: 'session.start', + payload: { liveSessionId: 'live_test' }, + }, + { ts: 250, seq: 2, type: 'tool.call', payload: { name: 'start_job' } }, + { ts: 400, seq: 3, type: 'session.end', payload: {} }, + ]); + const sequences = lines.map((line) => line.seq); + expect(sequences).toEqual([...sequences].sort((a, b) => a - b)); + }); + + it('creates a 0700 directory and a 0600 log file', async () => { + const base = await temporaryDirectory(); + const directory = join(base, 'nested', 'logs'); + const log = new SessionLog({ directory, liveSessionId: 'live_modes' }); + + log.write('session.start', {}); + await log.close(); + + if (process.platform !== 'win32') { + expect((await stat(directory)).mode & 0o777).toBe(0o700); + expect((await stat(log.filePath)).mode & 0o777).toBe(0o600); + } + await expect(readFile(log.filePath, 'utf8')).resolves.toContain( + 'session.start', + ); + }); + + it('rotates to a .1 file past maxBytes and keeps writing to a fresh file', async () => { + const base = await temporaryDirectory(); + const log = new SessionLog({ + directory: base, + liveSessionId: 'live_rotate', + maxBytes: 250, + now: () => 1, + }); + + // Two ~100-byte lines fit; the third overflows and triggers rotation. + log.write('backend.event', { marker: 'first', filler: 'x'.repeat(20) }); + log.write('backend.event', { marker: 'second', filler: 'x'.repeat(20) }); + log.write('backend.event', { marker: 'third', filler: 'x'.repeat(20) }); + await log.close(); + + const rotatedPath = `${log.filePath}.1`; + // The rotated-out stream flushes its remaining buffer asynchronously + // after the rename; poll until both files hold their final content. + await vi.waitFor(async () => { + const rotated = await readLines(rotatedPath); + expect(rotated.map((line) => line.payload['marker'])).toEqual([ + 'first', + 'second', + ]); + const current = await readLines(log.filePath); + expect(current.map((line) => line.payload['marker'])).toEqual(['third']); + expect(current[0]?.seq).toBe(3); + }); + }); + + it('does not throw when written after close', async () => { + const base = await temporaryDirectory(); + const log = new SessionLog({ directory: base, liveSessionId: 'live_late' }); + + log.write('session.start', {}); + await log.close(); + + // close() is terminal: a late write must not silently reopen a stream + // whose buffer would be lost at process.exit. Deleting the file first + // proves the write was a no-op rather than a reopen. + await rm(log.filePath); + expect(() => { + log.write('session.end', {}); + }).not.toThrow(); + await expect(stat(log.filePath)).rejects.toMatchObject({ code: 'ENOENT' }); + await log.close(); + }); + + it('flips to silent no-op mode after a write failure', async () => { + const base = await temporaryDirectory(); + const log = new SessionLog({ + directory: base, + liveSessionId: 'live_fail', + }); + const circular: Record = {}; + circular['self'] = circular; + + // JSON.stringify throws on the circular payload; the log must swallow it. + expect(() => { + log.write('error', circular); + }).not.toThrow(); + // Every later write is silently dropped. + expect(() => { + log.write('session.start', {}); + }).not.toThrow(); + await log.close(); + + await expect(stat(log.filePath)).rejects.toThrow(); + }); +}); diff --git a/packages/qwen-live/src/log/session-log.ts b/packages/qwen-live/src/log/session-log.ts new file mode 100644 index 00000000000..5fef462fe44 --- /dev/null +++ b/packages/qwen-live/src/log/session-log.ts @@ -0,0 +1,148 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Append-only JSONL log for one live session. This is the only durable + * record of a call (the harness no longer records live conversations), and + * doubles as the timing-bug forensics source: event names line up with the + * orchestrator's state machine. + * + * Redaction rules: never log API keys or raw audio; screenshots are logged + * as asset ids, not bytes. + */ + +import { + createWriteStream, + mkdirSync, + openSync, + renameSync, + type WriteStream, +} from 'node:fs'; +import { join } from 'node:path'; + +const DEFAULT_MAX_BYTES = 32 * 1024 * 1024; + +export type SessionLogEventType = + | 'session.start' + | 'session.end' + | 'vad.speech_started' + | 'vad.speech_stopped' + | 'transcript.user' + | 'transcript.assistant' + | 'response.created' + | 'response.done' + | 'response.cancelled' + | 'tool.call' + | 'tool.result' + | 'backend.event' + | 'inject.context' + | 'inject.speech' + | 'permission.request' + | 'permission.decision' + | 'playback.started' + | 'playback.cleared' + | 'error'; + +export interface SessionLogOptions { + directory: string; + liveSessionId: string; + maxBytes?: number; + now?: () => number; +} + +export class SessionLog { + private stream: WriteStream | undefined; + private readonly path: string; + private readonly maxBytes: number; + private readonly now: () => number; + private seq = 0; + private bytes = 0; + private rotations = 0; + private failed = false; + private closed = false; + + constructor(private readonly options: SessionLogOptions) { + this.path = join(options.directory, `${options.liveSessionId}.jsonl`); + this.maxBytes = options.maxBytes ?? DEFAULT_MAX_BYTES; + this.now = options.now ?? Date.now; + } + + /** + * Logging must never take the call down: failures flip to no-op mode, and + * writes after close() are dropped (a reopened stream would only lose its + * buffer at process.exit). + */ + write(type: SessionLogEventType, payload: Record): void { + if (this.failed || this.closed) return; + try { + const line = `${JSON.stringify({ + ts: this.now(), + seq: ++this.seq, + type, + payload, + })}\n`; + const bytes = Buffer.byteLength(line); + if (this.bytes + bytes > this.maxBytes) { + this.rotate(); + } + this.ensureStream().write(line); + this.bytes += bytes; + } catch { + this.failed = true; + } + } + + /** Terminal: later write() calls are documented no-ops. */ + async close(): Promise { + this.closed = true; + const stream = this.stream; + this.stream = undefined; + if (!stream) return; + await new Promise((resolve) => { + stream.end(() => { + resolve(); + }); + }); + } + + get filePath(): string { + return this.path; + } + + private ensureStream(): WriteStream { + if (!this.stream) { + mkdirSync(this.options.directory, { recursive: true, mode: 0o700 }); + // Open synchronously so the file exists on disk before write() returns; + // rotate() relies on a synchronous rename of an existing path. + const fd = openSync(this.path, 'a', 0o600); + this.stream = createWriteStream(this.path, { fd }); + this.stream.on('error', () => { + this.failed = true; + }); + } + return this.stream; + } + + /** + * Rotation must complete before the next line is written, or the freshly + * reopened stream would share the old inode and the rename would drag it + * along (observed as reordered lines in the rotated file and a missing + * main file). Rename synchronously first, then let the old stream flush + * its remaining buffer into the rotated file it already points at. + */ + private rotate(): void { + const stream = this.stream; + this.stream = undefined; + this.bytes = 0; + const rotatedTo = `${this.path}.${++this.rotations}`; + try { + renameSync(this.path, rotatedTo); + } catch { + /* nothing to rotate */ + } + stream?.end(); + } +} diff --git a/packages/qwen-live/src/logger.ts b/packages/qwen-live/src/logger.ts new file mode 100644 index 00000000000..0353066558a --- /dev/null +++ b/packages/qwen-live/src/logger.ts @@ -0,0 +1,48 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Minimal stderr logger. The repository bans `console`; a daemon still needs + * operator-facing diagnostics, and stderr is the right channel for them + * (stdout carries the machine-readable listening line only). + */ + +export type LogLevel = 'debug' | 'info' | 'warn' | 'error'; + +const LEVEL_ORDER: Record = { + debug: 10, + info: 20, + warn: 30, + error: 40, +}; + +export class LiveLogger { + constructor( + private readonly minLevel: LogLevel = (process.env[ + 'QWEN_LIVE_LOG_LEVEL' + ] as LogLevel | undefined) ?? 'info', + ) {} + + private write(level: LogLevel, message: string): void { + if (LEVEL_ORDER[level] < (LEVEL_ORDER[this.minLevel] ?? 20)) return; + process.stderr.write( + `[qwen-live] ${new Date().toISOString()} ${level.toUpperCase()} ${message}\n`, + ); + } + + debug(message: string): void { + this.write('debug', message); + } + info(message: string): void { + this.write('info', message); + } + warn(message: string): void { + this.write('warn', message); + } + error(message: string): void { + this.write('error', message); + } +} diff --git a/packages/qwen-live/src/orchestrator/injector.test.ts b/packages/qwen-live/src/orchestrator/injector.test.ts new file mode 100644 index 00000000000..5820d408d0f --- /dev/null +++ b/packages/qwen-live/src/orchestrator/injector.test.ts @@ -0,0 +1,414 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { Injector } from './injector.js'; +import type { InjectorItem, InjectorSink } from './injector.js'; + +const QUIET_GAP_MS = 800; + +class FakeSink implements InjectorSink { + contextCalls: string[] = []; + speechCalls: string[] = []; + injected: Array<{ item: InjectorItem; spoken: boolean }> = []; + contextResult = true; + speechResult = true; + + injectContext(text: string): boolean { + this.contextCalls.push(text); + return this.contextResult; + } + + injectSpeech(text: string): boolean { + this.speechCalls.push(text); + return this.speechResult; + } + + onInjected(item: InjectorItem, spoken: boolean): void { + this.injected.push({ item, spoken }); + } +} + +function complete(context: string, spoken?: string): InjectorItem { + return { kind: 'complete', context, ...(spoken ? { spoken } : {}) }; +} + +let sink: FakeSink; +let injector: Injector; + +function makeInjector(options?: { + quietGapMs?: number; + progressThrottleMs?: number; +}): Injector { + injector = new Injector({ + sink, + now: () => Date.now(), + ...(options?.quietGapMs !== undefined + ? { quietGapMs: options.quietGapMs } + : {}), + ...(options?.progressThrottleMs !== undefined + ? { progressThrottleMs: options.progressThrottleMs } + : {}), + }); + return injector; +} + +beforeEach(() => { + vi.useFakeTimers(); + // Far past the initial playbackDeadline (0) + quiet gap: window starts open. + vi.setSystemTime(1_000_000); + sink = new FakeSink(); + makeInjector(); +}); + +afterEach(() => { + injector.dispose(); + vi.useRealTimers(); +}); + +describe('Injector window conditions', () => { + it('holds items while the user is speaking and delivers on speech stop', () => { + injector.noteSpeechStarted(); + injector.enqueue(complete('tests passed')); + + expect(sink.contextCalls).toEqual([]); + expect(injector.pendingCount).toBe(1); + + injector.noteSpeechStopped(); + + expect(sink.contextCalls).toEqual(['tests passed']); + expect(injector.pendingCount).toBe(0); + }); + + it('holds items while a realtime response is in flight and delivers on done', () => { + injector.noteResponseCreated(); + injector.enqueue(complete('build finished')); + + expect(sink.contextCalls).toEqual([]); + expect(injector.pendingCount).toBe(1); + + injector.noteResponseDone(); + + expect(sink.contextCalls).toEqual(['build finished']); + }); + + it('waits out the estimated playback plus the quiet gap', () => { + // 48,000 bytes of 24 kHz mono PCM16 ≈ 1,000 ms of audio. + injector.noteOutputAudio(48_000); + injector.enqueue(complete('done')); + + expect(sink.contextCalls).toEqual([]); + + // One tick before playback end + quiet gap: still closed. + vi.advanceTimersByTime(1_000 + QUIET_GAP_MS - 1); + expect(sink.contextCalls).toEqual([]); + + vi.advanceTimersByTime(1); + expect(sink.contextCalls).toEqual(['done']); + }); + + it('stacks playback estimates for consecutive audio chunks', () => { + injector.noteOutputAudio(48_000); + injector.noteOutputAudio(48_000); + injector.enqueue(complete('done')); + + vi.advanceTimersByTime(2_000 + QUIET_GAP_MS - 1); + expect(sink.contextCalls).toEqual([]); + + vi.advanceTimersByTime(1); + expect(sink.contextCalls).toEqual(['done']); + }); + + it('reopens the window immediately when output audio is cleared', () => { + injector.noteOutputAudio(48_000 * 60); + injector.enqueue(complete('interrupted')); + vi.advanceTimersByTime(500); + expect(sink.contextCalls).toEqual([]); + + injector.noteOutputCleared(); + + expect(sink.contextCalls).toEqual(['interrupted']); + }); +}); + +describe('Injector batching', () => { + it('flushes a held batch as one context injection and one spoken line, in order', () => { + injector.noteResponseCreated(); + injector.enqueue({ + kind: 'complete', + context: 'job_1 finished', + spoken: 'Job one finished.', + jobHandle: 'job_1', + }); + injector.enqueue({ + kind: 'progress', + context: 'job_2 is halfway', + spoken: 'Job two is halfway.', + jobHandle: 'job_2', + }); + injector.enqueue({ + kind: 'permission', + context: 'job_3 wants to edit a file', + spoken: 'Job three needs permission.', + jobHandle: 'job_3', + requestId: 'req_1', + }); + expect(injector.pendingCount).toBe(3); + + injector.noteResponseDone(); + + expect(sink.contextCalls).toEqual([ + // Permission asks are moved to the front of the batch so the + // size-capped context join can never truncate their handles. + 'job_3 wants to edit a file\njob_1 finished\njob_2 is halfway', + ]); + expect(sink.speechCalls).toEqual([ + 'Job three needs permission. Job one finished. Job two is halfway.', + ]); + expect(sink.injected).toHaveLength(3); + expect(injector.pendingCount).toBe(0); + }); + + it('injects silently when no batch item carries a spoken line', () => { + injector.enqueue(complete('quiet update')); + + expect(sink.contextCalls).toEqual(['quiet update']); + expect(sink.speechCalls).toEqual([]); + expect(sink.injected).toEqual([ + { item: complete('quiet update'), spoken: false }, + ]); + }); +}); + +describe('Injector progress throttling', () => { + it('drops a second progress item for the same job inside the throttle window', () => { + injector.enqueue({ kind: 'progress', context: 'p1', jobHandle: 'job_1' }); + expect(sink.contextCalls).toEqual(['p1']); + + vi.advanceTimersByTime(60_000); + injector.enqueue({ kind: 'progress', context: 'p2', jobHandle: 'job_1' }); + + expect(sink.contextCalls).toEqual(['p1']); + expect(injector.pendingCount).toBe(0); + + // Past the 5-minute throttle the same job may report again. + vi.advanceTimersByTime(5 * 60_000); + injector.enqueue({ kind: 'progress', context: 'p3', jobHandle: 'job_1' }); + expect(sink.contextCalls).toEqual(['p1', 'p3']); + }); + + it('throttles per job handle, not globally', () => { + injector.enqueue({ kind: 'progress', context: 'a1', jobHandle: 'job_a' }); + vi.advanceTimersByTime(1_000); + injector.enqueue({ kind: 'progress', context: 'b1', jobHandle: 'job_b' }); + + expect(sink.contextCalls).toEqual(['a1', 'b1']); + }); + + it('keeps at most one queued progress item per job (newest wins)', () => { + makeInjector({ progressThrottleMs: 0 }); + injector.noteResponseCreated(); + injector.enqueue({ + kind: 'progress', + context: 'stale', + jobHandle: 'job_1', + }); + injector.enqueue({ + kind: 'progress', + context: 'fresh', + jobHandle: 'job_1', + }); + expect(injector.pendingCount).toBe(1); + + injector.noteResponseDone(); + + expect(sink.contextCalls).toEqual(['fresh']); + }); + + it('keys jobless progress on the full context, not a shared prefix', () => { + injector.noteResponseCreated(); + // Identical first 32 chars, diverging afterwards — the shape of two + // permission-retraction notices for one session. + const shared = '[BACKEND session_1] The permission request '; + injector.enqueue({ kind: 'progress', context: `${shared}(req_1) done.` }); + injector.enqueue({ kind: 'progress', context: `${shared}(req_2) done.` }); + expect(injector.pendingCount).toBe(2); + + injector.noteResponseDone(); + + expect(sink.contextCalls).toEqual([ + `${shared}(req_1) done.\n${shared}(req_2) done.`, + ]); + }); + + it('dedups true jobless duplicates to a single queued item', () => { + makeInjector({ progressThrottleMs: 0 }); + injector.noteResponseCreated(); + injector.enqueue({ kind: 'progress', context: 'same jobless notice' }); + injector.enqueue({ kind: 'progress', context: 'same jobless notice' }); + + expect(injector.pendingCount).toBe(1); + }); + + it('clears the throttle stamp when speech start drops queued progress', () => { + injector.noteResponseCreated(); + injector.enqueue({ kind: 'progress', context: 'p1', jobHandle: 'job_1' }); + expect(injector.pendingCount).toBe(1); + + // Barge-in drops the undelivered progress; the throttle window must not + // survive it, or the job goes silent for the whole window. + injector.noteSpeechStarted(); + expect(injector.pendingCount).toBe(0); + injector.noteSpeechStopped(); + injector.noteResponseDone(); + + vi.advanceTimersByTime(1_000); + injector.enqueue({ kind: 'progress', context: 'p2', jobHandle: 'job_1' }); + + expect(sink.contextCalls).toEqual(['p2']); + }); + + it('keeps the throttle stamp for progress that was actually delivered', () => { + injector.enqueue({ kind: 'progress', context: 'p1', jobHandle: 'job_1' }); + expect(sink.contextCalls).toEqual(['p1']); + + injector.noteSpeechStarted(); + injector.noteSpeechStopped(); + vi.advanceTimersByTime(1_000); + injector.enqueue({ kind: 'progress', context: 'p2', jobHandle: 'job_1' }); + + expect(sink.contextCalls).toEqual(['p1']); + }); +}); + +describe('Injector queue maintenance', () => { + it('drops queued progress on speech start but keeps conclusions and permission asks', () => { + injector.noteResponseCreated(); + injector.enqueue(complete('finished')); + injector.enqueue({ kind: 'progress', context: 'halfway', jobHandle: 'j' }); + injector.enqueue({ + kind: 'permission', + context: 'needs approval', + requestId: 'req_1', + }); + expect(injector.pendingCount).toBe(3); + + injector.noteSpeechStarted(); + expect(injector.pendingCount).toBe(2); + + injector.noteResponseDone(); + injector.noteSpeechStopped(); + expect(sink.contextCalls).toEqual(['needs approval\nfinished']); + }); + + it('retracts a queued permission ask by request id', () => { + injector.noteResponseCreated(); + injector.enqueue({ + kind: 'permission', + context: 'ask one', + requestId: 'req_1', + }); + injector.enqueue({ + kind: 'permission', + context: 'ask two', + requestId: 'req_2', + }); + + expect(injector.retractPermission('req_1')).toBe(true); + expect(injector.pendingCount).toBe(1); + expect(injector.retractPermission('req_unknown')).toBe(false); + + injector.noteResponseDone(); + expect(sink.contextCalls).toEqual(['ask two']); + }); + + it('dispose clears the queue and nothing is delivered afterwards', () => { + injector.noteResponseCreated(); + injector.enqueue(complete('stale one')); + injector.enqueue({ + kind: 'permission', + context: 'stale ask', + requestId: 'req_1', + }); + expect(injector.pendingCount).toBe(2); + + injector.dispose(); + + expect(injector.pendingCount).toBe(0); + injector.noteResponseDone(); + vi.advanceTimersByTime(60_000); + expect(sink.contextCalls).toEqual([]); + expect(sink.speechCalls).toEqual([]); + }); +}); + +describe('Injector transport refusal', () => { + it('requeues the batch and retries after the quiet gap when the sink refuses', () => { + sink.contextResult = false; + sink.speechResult = false; + + injector.enqueue(complete('important', 'Say this.')); + + expect(sink.contextCalls).toEqual(['important']); + expect(sink.speechCalls).toEqual(['Say this.']); + expect(sink.injected).toEqual([]); + expect(injector.pendingCount).toBe(1); + + sink.contextResult = true; + sink.speechResult = true; + vi.advanceTimersByTime(QUIET_GAP_MS); + + expect(sink.contextCalls).toEqual(['important', 'important']); + expect(sink.speechCalls).toEqual(['Say this.', 'Say this.']); + expect(sink.injected).toHaveLength(1); + expect(injector.pendingCount).toBe(0); + }); + + it('counts a spoken-only acceptance as delivered', () => { + sink.contextResult = false; + sink.speechResult = true; + + injector.enqueue(complete('body', 'Spoken line.')); + + expect(injector.pendingCount).toBe(0); + expect(sink.injected).toHaveLength(1); + }); +}); + +describe('Injector size caps', () => { + it('caps the combined context injection at 6000 chars', () => { + injector.noteSpeechStarted(); + injector.enqueue(complete('a'.repeat(4_000))); + injector.enqueue(complete('b'.repeat(4_000))); + injector.noteSpeechStopped(); + + expect(sink.contextCalls).toHaveLength(1); + expect(sink.contextCalls[0]).toHaveLength(6_000); + // The first item survives whole; the second is what the cap trims. + expect(sink.contextCalls[0]!.startsWith('a'.repeat(4_000))).toBe(true); + }); + + it('joins spoken lines whole and stops before breaching 280 chars', () => { + injector.noteSpeechStarted(); + injector.enqueue(complete('one', 'x'.repeat(200))); + injector.enqueue(complete('two', 'y'.repeat(200))); + injector.noteSpeechStopped(); + + expect(sink.speechCalls).toHaveLength(1); + // Whole-line greedy join: the second line would breach the cap, so + // only the first is spoken — never a mid-line cut of line two. + expect(sink.speechCalls[0]).toBe('x'.repeat(200)); + }); + + it('truncates a single over-long spoken line with an ellipsis', () => { + injector.noteSpeechStarted(); + injector.enqueue(complete('ctx', 'z'.repeat(400))); + injector.noteSpeechStopped(); + + expect(sink.speechCalls).toHaveLength(1); + expect(sink.speechCalls[0]).toBe(`${'z'.repeat(280)}…`); + }); +}); diff --git a/packages/qwen-live/src/orchestrator/injector.ts b/packages/qwen-live/src/orchestrator/injector.ts new file mode 100644 index 00000000000..ea64453cf70 --- /dev/null +++ b/packages/qwen-live/src/orchestrator/injector.ts @@ -0,0 +1,260 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Result backflow: takes normalized backend happenings and injects them into + * the realtime conversation at safe moments. + * + * Spoken/detail split: every item lands as silent context (the model can + * answer follow-ups from it), and speech-worthy items additionally trigger a + * short verbatim spoken line. + * + * The injection window is closed while any of these hold: + * 1. the user is speaking (VAD), + * 2. a realtime response is in flight, + * 3. output audio is still estimated to be playing. + * Protocol v6 has no playback acknowledgement from the Host, so (3) is a + * conservative estimate: bytes sent ÷ 48,000 B/s (24 kHz mono PCM16) plus a + * quiet gap. The v7 protocol upgrade replaces this with a real receipt. + */ + +const QUIET_GAP_MS = 800; +const RECHECK_MIN_MS = 100; +const PROGRESS_THROTTLE_MS = 5 * 60_000; +const MAX_SPOKEN_CHARS = 280; +const MAX_CONTEXT_CHARS = 6_000; +/** 24 kHz mono PCM16. */ +const PLAYBACK_BYTES_PER_MS = 48; + +export type InjectorItemKind = + | 'complete' + | 'progress' + | 'permission' + | 'error' + | 'speak'; + +export interface InjectorItem { + kind: InjectorItemKind; + /** Silent context body (without prefix). */ + context: string; + /** Verbatim spoken line; omitted items inject silently. */ + spoken?: string; + jobHandle?: string; + /** For permission items: lets a remote resolution retract the ask. */ + requestId?: string; +} + +export interface InjectorSink { + /** Silent context injection; false when the transport refused. */ + injectContext(text: string): boolean; + /** Verbatim speech request; false when the transport refused. */ + injectSpeech(text: string): boolean; + onInjected?(item: InjectorItem, spoken: boolean): void; +} + +export interface InjectorOptions { + sink: InjectorSink; + now?: () => number; + quietGapMs?: number; + progressThrottleMs?: number; +} + +/** + * Progress-throttle key: the job handle when the item has one, otherwise the + * item's full context (the map is per-call and bounded in practice). + */ +function progressKeyOf(item: InjectorItem): string { + return item.jobHandle ?? `ctx:${item.context}`; +} + +export class Injector { + private readonly sink: InjectorSink; + private readonly now: () => number; + private readonly quietGapMs: number; + private readonly progressThrottleMs: number; + + private queue: InjectorItem[] = []; + private speechInProgress = false; + private responseInFlight = false; + private playbackDeadline = 0; + private lastProgressAt = new Map(); + private timer: ReturnType | undefined; + private disposed = false; + + constructor(options: InjectorOptions) { + this.sink = options.sink; + this.now = options.now ?? Date.now; + this.quietGapMs = options.quietGapMs ?? QUIET_GAP_MS; + this.progressThrottleMs = + options.progressThrottleMs ?? PROGRESS_THROTTLE_MS; + } + + // -- window state signals (fed by the orchestrator) ---------------------- + + noteSpeechStarted(): void { + this.speechInProgress = true; + // Barge-in semantics: pending progress is stale the moment the user + // speaks; conclusions and permission asks stay queued. A dropped item + // was never delivered, so its throttle stamp must not stand — the job's + // next progress report may come well within the window. + const kept: InjectorItem[] = []; + for (const item of this.queue) { + if (item.kind === 'progress') { + this.lastProgressAt.delete(progressKeyOf(item)); + } else { + kept.push(item); + } + } + this.queue = kept; + } + + noteSpeechStopped(): void { + this.speechInProgress = false; + this.poke(); + } + + noteResponseCreated(): void { + this.responseInFlight = true; + } + + noteResponseDone(): void { + this.responseInFlight = false; + this.poke(); + } + + noteOutputAudio(bytes: number): void { + const start = Math.max(this.now(), this.playbackDeadline); + this.playbackDeadline = start + bytes / PLAYBACK_BYTES_PER_MS; + } + + noteOutputCleared(): void { + this.playbackDeadline = 0; + this.poke(); + } + + // -- queue -------------------------------------------------------------- + + enqueue(item: InjectorItem): void { + if (this.disposed) return; + if (item.kind === 'progress') { + // Throttle per job; jobless progress is keyed on its full context so + // distinct notices (which may share a long common prefix) never + // collide on one throttle window. + const key = progressKeyOf(item); + const last = this.lastProgressAt.get(key) ?? 0; + if (this.now() - last < this.progressThrottleMs) return; + this.lastProgressAt.set(key, this.now()); + // At most one queued progress item per key. + this.queue = this.queue.filter( + (queued) => + !(queued.kind === 'progress' && progressKeyOf(queued) === key), + ); + } + this.queue.push(item); + this.poke(); + } + + /** Retract a queued permission ask that was resolved elsewhere. */ + retractPermission(requestId: string): boolean { + const before = this.queue.length; + this.queue = this.queue.filter( + (item) => !(item.kind === 'permission' && item.requestId === requestId), + ); + return this.queue.length !== before; + } + + get pendingCount(): number { + return this.queue.length; + } + + dispose(): void { + this.disposed = true; + this.queue = []; + if (this.timer !== undefined) clearTimeout(this.timer); + this.timer = undefined; + } + + // -- delivery ----------------------------------------------------------- + + private windowClosedForMs(): number { + if (this.speechInProgress || this.responseInFlight) return -1; + const quietAt = this.playbackDeadline + this.quietGapMs; + const wait = quietAt - this.now(); + return wait > 0 ? wait : 0; + } + + private poke(): void { + if (this.disposed || this.queue.length === 0) return; + const wait = this.windowClosedForMs(); + if (wait < 0) return; // reopened by a state signal later + if (wait === 0) { + this.flush(); + return; + } + if (this.timer !== undefined) clearTimeout(this.timer); + this.timer = setTimeout( + () => { + this.timer = undefined; + this.poke(); + }, + Math.max(wait, RECHECK_MIN_MS), + ); + this.timer.unref?.(); + } + + private flush(): void { + if (this.queue.length === 0) return; + // Permission asks first: the context join is size-capped, and a + // truncated [PERMISSION] entry would lose the handle the model needs + // for respond_permission. + const batch = [ + ...this.queue.filter((item) => item.kind === 'permission'), + ...this.queue.filter((item) => item.kind !== 'permission'), + ]; + this.queue = []; + + // One combined silent context injection for the whole batch. + const context = batch + .map((item) => item.context) + .join('\n') + .slice(0, MAX_CONTEXT_CHARS); + const contextAccepted = this.sink.injectContext(context); + + // One combined spoken line for the speech-worthy items — whole lines + // only, since the model is told to read the text verbatim. + const spokenLines = batch + .map((item) => item.spoken) + .filter((line): line is string => typeof line === 'string' && !!line); + let spoken = ''; + for (const line of spokenLines) { + const candidate = spoken ? `${spoken} ${line}` : line; + if (candidate.length > MAX_SPOKEN_CHARS && spoken) break; + spoken = + candidate.length > MAX_SPOKEN_CHARS + ? `${candidate.slice(0, MAX_SPOKEN_CHARS)}…` + : candidate; + } + let spokenAccepted = false; + if (spoken) { + spokenAccepted = this.sink.injectSpeech(spoken); + } + + if (!contextAccepted && !spokenAccepted) { + // Transport refused (socket busy/closed): requeue and retry shortly. + this.queue = [...batch, ...this.queue]; + if (this.timer !== undefined) clearTimeout(this.timer); + this.timer = setTimeout(() => { + this.timer = undefined; + this.poke(); + }, this.quietGapMs); + this.timer.unref?.(); + return; + } + for (const item of batch) { + this.sink.onInjected?.(item, spokenLines.length > 0); + } + } +} diff --git a/packages/qwen-live/src/orchestrator/live-session.test.ts b/packages/qwen-live/src/orchestrator/live-session.test.ts new file mode 100644 index 00000000000..38906c8a5ed --- /dev/null +++ b/packages/qwen-live/src/orchestrator/live-session.test.ts @@ -0,0 +1,952 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; +import type { + BackendAdaptor, + BackendCapabilities, + BackendEvent, + BackendHandle, + ContentBlock, + PermissionDecision, + PermissionOption, + PromptReceipt, + SessionSummary, +} from '../adaptor/types.js'; +import type { LiveScreenContextCapture } from '../host/live-host-coordinator.js'; +import type { LiveState } from '../host/types.js'; +import type { SessionLog } from '../log/session-log.js'; +import type { + openQwenRealtimeSession, + QwenRealtimeCallbacks, + QwenRealtimeConfig, + QwenRealtimeSession, + RealtimeCloseInfo, + RealtimeCloseOptions, + RealtimeFunctionCallRef, + RealtimeTranscriptEntry, +} from '../realtime/realtime-session.js'; +import { LIVE_SESSION_TOOLS } from '../tools/definitions.js'; +import { LiveSession } from './live-session.js'; + +const PERMISSION_OPTIONS: readonly PermissionOption[] = [ + { optionId: 'allow', kind: 'proceed' }, + { optionId: 'deny', kind: 'reject' }, +]; + +const delay = (ms: number): Promise => + new Promise((resolve) => { + setTimeout(resolve, ms); + }); + +// -- test doubles ----------------------------------------------------------- + +/** Minimal push-driven async queue backing FakeAdaptor.events(). */ +class AsyncQueue implements AsyncIterable { + private readonly buffered: T[] = []; + private readonly waiters: Array<(result: IteratorResult) => void> = []; + private ended = false; + + push(item: T): void { + const waiter = this.waiters.shift(); + if (waiter) { + waiter({ value: item, done: false }); + return; + } + this.buffered.push(item); + } + + end(): void { + this.ended = true; + for (const waiter of this.waiters.splice(0)) { + waiter({ value: undefined as never, done: true }); + } + } + + [Symbol.asyncIterator](): AsyncIterator { + return { + next: (): Promise> => { + if (this.buffered.length > 0) { + const value = this.buffered.shift() as T; + return Promise.resolve({ value, done: false }); + } + if (this.ended) { + return Promise.resolve({ value: undefined as never, done: true }); + } + return new Promise((resolve) => { + this.waiters.push(resolve); + }); + }, + }; + } +} + +class FakeAdaptor implements BackendAdaptor { + readonly name = 'fake'; + busy = false; + promptReceipt: PromptReceipt = { status: 'accepted', jobRef: 'p1' }; + summaries: SessionSummary[] = []; + readonly queues = new Map>(); + private sessionSeq = 0; + + readonly createSession = vi.fn( + async (_opts?: { + cwd?: string; + label?: string; + }): Promise => ({ + id: `s${++this.sessionSeq}`, + adaptor: this.name, + }), + ); + + readonly prompt = vi.fn( + async ( + _handle: BackendHandle, + _blocks: readonly ContentBlock[], + _opts?: { steer?: boolean }, + ): Promise => this.promptReceipt, + ); + + readonly cancel = vi.fn(async (_handle: BackendHandle): Promise => {}); + + readonly respondPermission = vi.fn( + async ( + _handle: BackendHandle, + _requestId: string, + _decision: PermissionDecision, + ): Promise<'delivered' | 'already_resolved'> => 'delivered', + ); + + capabilities(): BackendCapabilities { + return { + steering: 'native', + imageInput: true, + permissionForwarding: true, + proactiveSpeak: true, + sessionList: true, + eventDelivery: 'stream', + }; + } + + async preflight(): Promise {} + + async listSessions(): Promise { + return this.summaries; + } + + events(handle: BackendHandle): AsyncIterable { + return this.queue(handle.id); + } + + isBusy(_handle: BackendHandle): boolean { + return this.busy; + } + + async close(): Promise {} + + queue(backendId: string): AsyncQueue { + let queue = this.queues.get(backendId); + if (!queue) { + queue = new AsyncQueue(); + this.queues.set(backendId, queue); + } + return queue; + } +} + +/** + * FakeAdaptor whose events() hands out one stream per subscription: the + * first can be ended (without session_closed) to simulate a dropped SSE + * stream; the pump must resubscribe and land on the second. + */ +class ResubscribeAdaptor extends FakeAdaptor { + readonly streams = [ + new AsyncQueue(), + new AsyncQueue(), + ]; + eventsCalls = 0; + + override events(): AsyncIterable { + const stream = this.streams[ + Math.min(this.eventsCalls, this.streams.length - 1) + ] as AsyncQueue; + this.eventsCalls += 1; + return stream; + } +} + +function createFakeHost(capture: LiveScreenContextCapture) { + const states: Array> = []; + return { + states, + setCallState: vi.fn( + ( + _epoch: number, + state: Exclude, + ): boolean => { + states.push(state); + return true; + }, + ), + setCoordinator: vi.fn( + ( + _epoch: number, + _locator: { workspaceCwd: string; sessionId: string }, + ): boolean => true, + ), + sendOutputAudio: vi.fn( + (_epoch: number, _pcm16: Uint8Array): boolean => true, + ), + clearOutput: vi.fn((_epoch: number): void => {}), + setCaption: vi.fn((_epoch: number, _caption: string): boolean => true), + setStatusText: vi.fn( + (_epoch: number, _statusText?: string): boolean => true, + ), + setTranscript: vi.fn( + (_epoch: number, _transcript: string): boolean => true, + ), + failCall: vi.fn((_epoch: number, _message?: string): boolean => true), + captureScreenContext: vi.fn( + async (_callerSessionId: string): Promise => + capture, + ), + }; +} + +function createFakeRealtime() { + return { + callEpoch: 1, + closed: new Promise(() => {}), + pushAudio: vi.fn((_pcm16: Uint8Array): boolean => true), + commitInputAudio: vi.fn((): boolean => true), + clearInputAudio: vi.fn((): boolean => true), + cancelResponse: vi.fn((): boolean => true), + submitFunctionOutput: vi.fn( + (_ref: RealtimeFunctionCallRef, _output: string): boolean => true, + ), + sendBackendContext: vi.fn((_text: string): boolean => true), + speakToUser: vi.fn((_message: string): boolean => true), + takeTranscriptTail: vi.fn((): readonly RealtimeTranscriptEntry[] => []), + close: vi.fn((_options?: RealtimeCloseOptions): void => {}), + }; +} + +type FakeRealtime = ReturnType; + +// -- rig --------------------------------------------------------------------- + +let tempDir: string; +let pngPath: string; + +beforeAll(async () => { + tempDir = await mkdtemp(join(tmpdir(), 'qwen-live-session-test-')); + pngPath = join(tempDir, 'shot.png'); + // A real (if tiny) PNG signature so image blocks carry non-empty bytes. + await writeFile( + pngPath, + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 1, 2, 3, 4]), + ); +}); + +afterAll(async () => { + await rm(tempDir, { recursive: true, force: true }); +}); + +interface Rig { + session: LiveSession; + adaptor: FakeAdaptor; + logWrites: ReturnType; + host: ReturnType; + realtime: FakeRealtime; + config: QwenRealtimeConfig; + callbacks: QwenRealtimeCallbacks; +} + +async function startSession(adaptorArg?: FakeAdaptor): Promise { + const adaptor = adaptorArg ?? new FakeAdaptor(); + const host = createFakeHost({ + appName: 'Safari', + windowTitle: 'Docs', + accessibilityText: 'visible text', + screenshotPath: pngPath, + }); + const realtime = createFakeRealtime(); + const logWrites = vi.fn(); + let config: QwenRealtimeConfig | undefined; + let callbacks: QwenRealtimeCallbacks = {}; + const openRealtime: typeof openQwenRealtimeSession = (cfg, cbs = {}) => { + config = cfg; + callbacks = cbs; + return Promise.resolve(realtime as unknown as QwenRealtimeSession); + }; + const session = new LiveSession({ + host, + adaptor, + realtime: { + endpoint: 'https://dashscope.example.com', + model: 'qwen-omni-turbo-realtime', + voice: 'Cherry', + }, + log: { write: logWrites, close: async () => {} } as unknown as SessionLog, + openRealtime, + }); + await session.start({ epoch: 1, callId: 'call-1', mode: 'new' }); + if (!config) throw new Error('openRealtime was not called'); + return { + session, + adaptor, + host, + realtime, + config, + callbacks, + logWrites, + }; +} + +let callSeq = 0; + +function callTool( + callbacks: QwenRealtimeCallbacks, + name: string, + args: Record, + activeTranscript: readonly RealtimeTranscriptEntry[] = [], +): void { + callSeq += 1; + callbacks.onFunctionCall?.({ + callEpoch: 1, + responseId: `resp_${callSeq}`, + callId: `fc_${callSeq}`, + name, + arguments: JSON.stringify(args), + activeTranscript, + }); +} + +function receipts(realtime: FakeRealtime): Array> { + return realtime.submitFunctionOutput.mock.calls.map( + ([, output]) => JSON.parse(output) as Record, + ); +} + +async function awaitReceipts( + realtime: FakeRealtime, + count: number, +): Promise>> { + await vi.waitFor(() => { + expect(realtime.submitFunctionOutput).toHaveBeenCalledTimes(count); + }); + return receipts(realtime); +} + +// -- tests -------------------------------------------------------------------- + +describe('LiveSession', () => { + it('start opens the realtime session with the live tool surface and walks starting → listening', async () => { + const { config, host } = await startSession(); + + expect(config.tools).toBe(LIVE_SESSION_TOOLS); + expect(config.instructions.length).toBeGreaterThan(0); + expect(host.states).toEqual(['starting', 'listening']); + }); + + it('handoff creates a default session and prompts with the task plus voice context', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: 'fix tests' }, [ + { role: 'user', text: 'please fix the failing tests' }, + ]); + const [receipt] = await awaitReceipts(realtime, 1); + + expect(adaptor.createSession).toHaveBeenCalledTimes(1); + expect(adaptor.prompt).toHaveBeenCalledTimes(1); + const blocks = adaptor.prompt.mock.calls[0]?.[1]; + expect(blocks).toBeDefined(); + const text = blocks?.[0]; + if (text?.type !== 'text') throw new Error('expected a leading text block'); + expect(text.text).toContain('fix tests'); + expect(text.text).toContain(''); + expect(text.text).toContain('please fix the failing tests'); + expect(receipt).toMatchObject({ + status: 'accepted', + job: 'job_1', + session: 'session_1', + }); + }); + + it('handoff to a busy session steers instead of prompting fresh', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + adaptor.busy = true; + + callTool(callbacks, 'handoff', { task: 'also run lint' }); + await awaitReceipts(realtime, 1); + + expect(adaptor.prompt.mock.calls[0]?.[2]).toEqual({ steer: true }); + }); + + it('a steer that joined the running turn reuses that job instead of orphaning it', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: 'run the tests' }); + const [first] = await awaitReceipts(realtime, 1); + expect(first).toMatchObject({ job: 'job_1', session: 'session_1' }); + adaptor.queue('s1').push({ type: 'turn_started', jobRef: 'p1' }); + + // The backend joins the steer to the running turn: same jobRef back. + adaptor.busy = true; + adaptor.promptReceipt = { + status: 'accepted', + jobRef: 'p1', + joinedActiveTurn: true, + note: 'joined the currently running task', + }; + callTool(callbacks, 'handoff', { task: 'also run lint' }); + const [, second] = await awaitReceipts(realtime, 2); + expect(second).toMatchObject({ job: 'job_1', session: 'session_1' }); + + // One job only — and the turn's completion retires it. + adaptor.queue('s1').push({ + type: 'turn_complete', + jobRef: 'p1', + summary: 'all done.', + }); + await vi.waitFor(() => { + expect(realtime.sendBackendContext).toHaveBeenCalledTimes(1); + }); + adaptor.busy = false; + callTool(callbacks, 'session_monitor', { session: 'session_1' }); + const [, , monitor] = await awaitReceipts(realtime, 3); + expect(monitor).toEqual({ + status: 'ok', + session: 'session_1', + state: 'idle', + }); + }); + + it('handoff attaches appshot-registered assets as image blocks', async () => { + const { adaptor, callbacks, host, realtime } = await startSession(); + + callTool(callbacks, 'appshot', {}); + const [appshotReceipt] = await awaitReceipts(realtime, 1); + expect(host.captureScreenContext).toHaveBeenCalledTimes(1); + expect(appshotReceipt).toMatchObject({ + status: 'ok', + app: 'Safari', + window: 'Docs', + asset: 'asset_1', + }); + + callTool(callbacks, 'handoff', { + task: 'describe this window', + input_refs: ['asset_1'], + }); + await awaitReceipts(realtime, 2); + + const blocks = adaptor.prompt.mock.calls[0]?.[1]; + expect(blocks).toHaveLength(2); + const image = blocks?.[1]; + if (image?.type !== 'image') throw new Error('expected an image block'); + expect(image.mimeType).toBe('image/png'); + expect(image.data.byteLength).toBeGreaterThan(0); + }); + + it('session_list returns handles and states for backend sessions', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + adaptor.summaries = [ + { + handle: { id: 'a', adaptor: 'fake' }, + label: 'One', + cwd: '/tmp/a', + state: 'idle', + }, + { handle: { id: 'b', adaptor: 'fake' }, state: 'busy' }, + ]; + + callTool(callbacks, 'session_list', {}); + const [receipt] = await awaitReceipts(realtime, 1); + + expect(receipt).toMatchObject({ status: 'ok' }); + expect(receipt?.['sessions']).toEqual([ + { handle: 'session_1', label: 'One', cwd: '/tmp/a', state: 'idle' }, + { handle: 'session_2', state: 'busy' }, + ]); + }); + + it('session_stop cancels the backend turn and marks the job cancelled', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: 'long task' }); + await awaitReceipts(realtime, 1); + + callTool(callbacks, 'session_stop', { job: 'job_1' }); + const [, stopReceipt] = await awaitReceipts(realtime, 2); + + expect(adaptor.cancel).toHaveBeenCalledTimes(1); + expect(stopReceipt).toMatchObject({ + status: 'cancelling', + session: 'session_1', + }); + + callTool(callbacks, 'session_monitor', { job: 'job_1' }); + const [, , monitorReceipt] = await awaitReceipts(realtime, 3); + expect(monitorReceipt).toMatchObject({ + job: 'job_1', + job_state: 'cancelled', + }); + }); + + it('injects turn_complete and turn_error events as context plus speech', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: 'run the tests' }); + await awaitReceipts(realtime, 1); + const queue = adaptor.queue('s1'); + + queue.push({ + type: 'turn_complete', + jobRef: 'p1', + summary: 'done: all tests pass', + }); + await vi.waitFor(() => { + expect(realtime.sendBackendContext).toHaveBeenCalledTimes(1); + }); + expect(realtime.sendBackendContext.mock.calls[0]?.[0]).toMatch( + /^\[COMPLETE job_1\]/, + ); + expect(realtime.speakToUser).toHaveBeenCalledTimes(1); + expect(realtime.speakToUser.mock.calls[0]?.[0]).toContain('job_1'); + + queue.push({ type: 'turn_error', jobRef: 'p1', error: 'lint exploded' }); + await vi.waitFor(() => { + expect(realtime.sendBackendContext).toHaveBeenCalledTimes(2); + }); + expect(realtime.sendBackendContext.mock.calls[1]?.[0]).toMatch( + /^\[ERROR job_1\]/, + ); + }); + + it('routes permission requests to the voice and relays the answer back', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: 'clean tmp' }); + await awaitReceipts(realtime, 1); + const queue = adaptor.queue('s1'); + + queue.push({ + type: 'permission_request', + requestId: 'r1', + title: 'Bash: rm -rf /tmp', + options: PERMISSION_OPTIONS, + }); + await vi.waitFor(() => { + expect(realtime.sendBackendContext).toHaveBeenCalledTimes(1); + }); + expect(realtime.sendBackendContext.mock.calls[0]?.[0]).toContain( + '[PERMISSION req_1]', + ); + expect(realtime.speakToUser).toHaveBeenCalledTimes(1); + + callTool(callbacks, 'respond_permission', { + request_id: 'req_1', + decision: 'allow', + }); + const [, respondReceipt] = await awaitReceipts(realtime, 2); + + expect(adaptor.respondPermission).toHaveBeenCalledTimes(1); + const [, requestId, decision] = + adaptor.respondPermission.mock.calls[0] ?? []; + expect(requestId).toBe('r1'); + expect(decision).toBe('allow'); + expect(respondReceipt).toEqual({ status: 'delivered' }); + }); + + it('relays a respond_permission note to the backend session after the vote', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: 'clean tmp' }); + await awaitReceipts(realtime, 1); + adaptor.queue('s1').push({ + type: 'permission_request', + requestId: 'r1', + title: 'Bash: rm -rf /tmp', + options: PERMISSION_OPTIONS, + }); + await vi.waitFor(() => { + expect(realtime.sendBackendContext).toHaveBeenCalledTimes(1); + }); + + adaptor.busy = true; + callTool(callbacks, 'respond_permission', { + request_id: 'req_1', + decision: 'allow', + note: 'only the cache subfolder', + }); + const [, respondReceipt] = await awaitReceipts(realtime, 2); + + expect(respondReceipt).toEqual({ status: 'delivered' }); + expect(adaptor.respondPermission).toHaveBeenCalledWith( + expect.objectContaining({ id: 's1' }), + 'r1', + 'allow', + ); + // The vote channel has no note field: the constraint rides the prompt + // path to the same session, steered into the running turn. + expect(adaptor.prompt).toHaveBeenCalledTimes(2); + const [handle, blocks, opts] = adaptor.prompt.mock.calls[1] ?? []; + expect(handle).toEqual(expect.objectContaining({ id: 's1' })); + const text = blocks?.[0]; + if (text?.type !== 'text') throw new Error('expected a text block'); + expect(text.text).toContain('only the cache subfolder'); + expect(text.text).toContain('Bash: rm -rf /tmp'); + expect(opts).toEqual({ steer: true }); + }); + + it('delivers a note-less respond_permission without a follow-up prompt', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: 'clean tmp' }); + await awaitReceipts(realtime, 1); + adaptor.queue('s1').push({ + type: 'permission_request', + requestId: 'r1', + title: 'Bash: rm -rf /tmp', + options: PERMISSION_OPTIONS, + }); + await vi.waitFor(() => { + expect(realtime.sendBackendContext).toHaveBeenCalledTimes(1); + }); + + callTool(callbacks, 'respond_permission', { + request_id: 'req_1', + decision: 'deny', + }); + const [, respondReceipt] = await awaitReceipts(realtime, 2); + + expect(respondReceipt).toEqual({ status: 'delivered' }); + // Only the handoff prompt — no constraint relay was needed. + expect(adaptor.prompt).toHaveBeenCalledTimes(1); + }); + + it('retracts a queued permission ask resolved elsewhere before injection', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: 'clean tmp' }); + await awaitReceipts(realtime, 1); + const queue = adaptor.queue('s1'); + + // Close the injection window: a realtime response is in flight. + callbacks.onResponseCreated?.({ + callEpoch: 1, + responseId: 'resp_open', + authority: 'direct', + }); + + queue.push({ + type: 'permission_request', + requestId: 'r2', + title: 'Bash: rm -rf /tmp', + options: PERMISSION_OPTIONS, + }); + await delay(30); + expect(realtime.sendBackendContext).not.toHaveBeenCalled(); + + // Resolved from the WebShell before the voice ever asked. + queue.push({ type: 'permission_resolved', requestId: 'r2', byUs: false }); + await delay(30); + + // Reopen the window: the retracted ask must not surface. + callbacks.onResponseDone?.({ callEpoch: 1, responseId: 'resp_open' }); + await delay(50); + expect(realtime.sendBackendContext).not.toHaveBeenCalled(); + expect(realtime.speakToUser).not.toHaveBeenCalled(); + }); + + it('barge-in clears the host output', async () => { + const { callbacks, host } = await startSession(); + + callbacks.onBargeIn?.({ callEpoch: 1, responseId: 'resp_x' }); + + expect(host.clearOutput).toHaveBeenCalledTimes(1); + }); + + it('stop resolves immediately when no response is in flight', async () => { + const { realtime, session } = await startSession(); + + const outcome = await session.stop({ epoch: 1, callId: 'call-1' }); + + expect(outcome).toBeUndefined(); + expect(realtime.close).toHaveBeenCalledTimes(1); + }); + + it('stop waits for the in-flight response to settle', async () => { + const { callbacks, realtime, session } = await startSession(); + + callbacks.onResponseCreated?.({ + callEpoch: 1, + responseId: 'resp_1', + authority: 'direct', + }); + + let settled = false; + const pending = session + .stop({ epoch: 1, callId: 'call-1' }) + .then((outcome) => { + settled = true; + return outcome; + }); + await delay(150); + expect(settled).toBe(false); + + callbacks.onResponseDone?.({ callEpoch: 1, responseId: 'resp_1' }); + await vi.waitFor( + () => { + expect(settled).toBe(true); + }, + { timeout: 3_000, interval: 100 }, + ); + expect(await pending).toBeUndefined(); + expect(realtime.close).toHaveBeenCalledTimes(1); + }); + + it('stop drain settles on the input-commit ack instead of burning the budget', async () => { + // Default drain budget: 30 s. The commit ack must settle the stop in + // milliseconds — resolving only at the deadline is the bug. + const { callbacks, realtime, session } = await startSession(); + + callbacks.onSpeechStarted?.({ callEpoch: 1 }); + + let settled = false; + const pending = session + .stop({ epoch: 1, callId: 'call-1' }) + .then((outcome) => { + settled = true; + return outcome; + }); + expect(realtime.commitInputAudio).toHaveBeenCalledTimes(1); + await delay(150); + expect(settled).toBe(false); + + callbacks.onInputCommitted?.({ callEpoch: 1 }); + await vi.waitFor( + () => { + expect(settled).toBe(true); + }, + { timeout: 2_000, interval: 50 }, + ); + expect(await pending).toBeUndefined(); + expect(realtime.close).toHaveBeenCalledTimes(1); + }); + + it('stop fails fast when the trailing speech cannot be committed', async () => { + const { callbacks, realtime, session } = await startSession(); + realtime.commitInputAudio.mockReturnValue(false); + + callbacks.onSpeechStarted?.({ callEpoch: 1 }); + const outcome = await session.stop({ epoch: 1, callId: 'call-1' }); + + expect(outcome).toEqual({ + error: 'Live Voice could not commit the final spoken input.', + }); + }); + + it('keeps the call state at stopping when response.created arrives during the drain', async () => { + const { callbacks, host, session } = await startSession(); + + callbacks.onSpeechStarted?.({ callEpoch: 1 }); + let settled = false; + const pending = session + .stop({ epoch: 1, callId: 'call-1' }) + .then((outcome) => { + settled = true; + return outcome; + }); + callbacks.onInputCommitted?.({ callEpoch: 1 }); + + // semantic_vad create_response: the committed trailing speech spawns a + // response mid-drain. It must hold the drain open, but never flip the + // coordinator back to 'speaking' — that would strand the stopping call. + callbacks.onResponseCreated?.({ + callEpoch: 1, + responseId: 'resp_tail', + authority: 'direct', + }); + expect(host.states[host.states.length - 1]).toBe('stopping'); + await delay(150); + expect(settled).toBe(false); + + callbacks.onResponseDone?.({ callEpoch: 1, responseId: 'resp_tail' }); + await vi.waitFor( + () => { + expect(settled).toBe(true); + }, + { timeout: 2_000, interval: 50 }, + ); + expect(await pending).toBeUndefined(); + expect(host.states).not.toContain('speaking'); + }); + + it('speaks error strings with mid-token periods untruncated', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: 'read config' }); + await awaitReceipts(realtime, 1); + const queue = adaptor.queue('s1'); + + queue.push({ + type: 'turn_error', + jobRef: 'p1', + error: 'ENOENT: open /home/user/.qwen-live/config.json', + }); + await vi.waitFor(() => { + expect(realtime.speakToUser).toHaveBeenCalledTimes(1); + }); + expect(realtime.speakToUser.mock.calls[0]?.[0]).toContain( + '/home/user/.qwen-live/config.json', + ); + + queue.push({ + type: 'turn_error', + jobRef: 'p1', + error: 'Connection refused: 10.0.0.1:4170', + }); + await vi.waitFor(() => { + expect(realtime.speakToUser).toHaveBeenCalledTimes(2); + }); + expect(realtime.speakToUser.mock.calls[1]?.[0]).toContain('10.0.0.1:4170'); + }); + + it('speaks the closing sentence of a long CJK summary', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: '跑测试' }); + await awaitReceipts(realtime, 1); + + // >200 chars, multiple sentences, no whitespace after 。 (standard CJK + // typography): the spoken line must be the LAST sentence, complete. + const body = `${'任务进行中'.repeat(50)}。`; + const closing = '所有测试都通过了。'; + adaptor.queue('s1').push({ + type: 'turn_complete', + jobRef: 'p1', + summary: `${body}${closing}`, + }); + await vi.waitFor(() => { + expect(realtime.speakToUser).toHaveBeenCalledTimes(1); + }); + expect(realtime.speakToUser.mock.calls[0]?.[0]).toBe( + `Task job_1 finished. ${closing}`, + ); + }); + + it('resubscribes after the event stream ends without session_closed', async () => { + const adaptor = new ResubscribeAdaptor(); + const { callbacks, realtime } = await startSession(adaptor); + + callTool(callbacks, 'handoff', { task: 'long task' }); + await awaitReceipts(realtime, 1); + expect(adaptor.eventsCalls).toBe(1); + + // The stream drops without a session_closed (daemon restart, broken + // SSE connection) — the session must not go permanently unobserved. + adaptor.streams[0]?.end(); + adaptor.streams[1]?.push({ + type: 'turn_complete', + jobRef: 'p1', + summary: 'all tests pass.', + }); + + await vi.waitFor( + () => { + expect(realtime.sendBackendContext).toHaveBeenCalledTimes(1); + }, + { timeout: 5_000, interval: 100 }, + ); + expect(realtime.sendBackendContext.mock.calls[0]?.[0]).toMatch( + /^\[COMPLETE job_1\]/, + ); + expect(adaptor.eventsCalls).toBe(2); + }, 10_000); + + it('pushAudio forwards frames to realtime but not while stopping', async () => { + const { callbacks, realtime, session } = await startSession(); + + const frame = Buffer.from([1, 2]); + expect( + session.pushAudio({ epoch: 1, callId: 'call-1', pcm16: frame }), + ).toBe(true); + expect(realtime.pushAudio).toHaveBeenCalledTimes(1); + expect(realtime.pushAudio).toHaveBeenCalledWith(frame); + + callbacks.onResponseCreated?.({ + callEpoch: 1, + responseId: 'resp_1', + authority: 'direct', + }); + const stopPending = session.stop({ epoch: 1, callId: 'call-1' }); + + expect( + session.pushAudio({ epoch: 1, callId: 'call-1', pcm16: frame }), + ).toBe(true); + expect(realtime.pushAudio).toHaveBeenCalledTimes(1); + + callbacks.onResponseDone?.({ callEpoch: 1, responseId: 'resp_1' }); + await stopPending; + }); + + it('rebuilds the default session after it closes mid-call', async () => { + const { adaptor, callbacks, realtime, logWrites } = await startSession(); + + // First handoff creates and uses the default session… + callTool(callbacks, 'handoff', { task: 'first task' }); + await awaitReceipts(realtime, 1); + const firstCreateCalls = adaptor.createSession.mock.calls.length; + + // …then that session closes (WebShell deletion / daemon restart). + // The pump consumes the event asynchronously; wait until the closure + // has been observed before handing off again. + adaptor.queue('s1').push({ type: 'session_closed' }); + adaptor.queue('s1').end(); + await vi.waitFor(() => { + expect(logWrites).toHaveBeenCalledWith( + 'backend.event', + expect.objectContaining({ type: 'session_closed' }), + ); + }); + + // A session-less handoff must NOT re-target the dead session. + callTool(callbacks, 'handoff', { task: 'second task' }); + await awaitReceipts(realtime, 2); + expect(adaptor.createSession.mock.calls.length).toBeGreaterThan( + firstCreateCalls, + ); + const targets = adaptor.prompt.mock.calls.map( + (call) => (call[0] as { id: string }).id, + ); + expect(new Set(targets).size).toBe(targets.length); + }); + + it('reconciles stale running jobs when the backend reports idle', async () => { + const { adaptor, callbacks, realtime } = await startSession(); + + callTool(callbacks, 'handoff', { task: 'a task' }); + await awaitReceipts(realtime, 1); + // The turn_complete is emitted while no one consumes it (pump aborted + // between calls); mark the adaptor idle and list sessions. + adaptor.busy = false; + adaptor.summaries = [ + { handle: { id: 's1', adaptor: 'fake' }, state: 'idle' }, + ]; + callTool(callbacks, 'session_list', {}); + await awaitReceipts(realtime, 2); + const list = receipts(realtime)[1]; + // The stale running job must not survive the idle reconciliation. + expect(list?.['sessions']).toEqual([ + { handle: 'session_1', state: 'idle' }, + ]); + }); +}); diff --git a/packages/qwen-live/src/orchestrator/live-session.ts b/packages/qwen-live/src/orchestrator/live-session.ts new file mode 100644 index 00000000000..47ed394d240 --- /dev/null +++ b/packages/qwen-live/src/orchestrator/live-session.ts @@ -0,0 +1,1140 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * One live call, end to end: owns the realtime connection, routes audio + * between the Host and the provider, dispatches the seven orchestration + * tools, pumps backend events into the injector, and drains gracefully on + * stop. + * + * The state-machine disciplines are ported from qwen-code's + * live-session-coordinator (epoch + generation fences, bounded stop drain); + * the backend seam is the BackendAdaptor port instead of the in-process + * bridge. + */ + +import { readFile } from 'node:fs/promises'; +import type { + BackendAdaptor, + BackendEvent, + BackendHandle, + ContentBlock, +} from '../adaptor/types.js'; +import type { LiveScreenContextCapture } from '../host/live-host-coordinator.js'; +import type { LiveState } from '../host/types.js'; +import { buildLiveInstructions } from '../realtime/instructions.js'; +import { + openQwenRealtimeSession, + QWEN_REALTIME_LIMITS, + type QwenRealtimeSession, + type RealtimeFunctionCall, + type RealtimeTranscriptEntry, +} from '../realtime/realtime-session.js'; +import type { SessionLog } from '../log/session-log.js'; +import { PermissionBroker } from '../permissions/permission-broker.js'; +import { + APPSHOT_TOOL_NAME, + HANDOFF_TOOL_NAME, + LIVE_SESSION_TOOLS, + RESPOND_PERMISSION_TOOL_NAME, + SESSION_CREATE_TOOL_NAME, + SESSION_LIST_TOOL_NAME, + SESSION_MONITOR_TOOL_NAME, + SESSION_STOP_TOOL_NAME, +} from '../tools/definitions.js'; +import { + ToolDispatcher, + type ToolContext, + type ToolHandler, +} from '../tools/dispatcher.js'; +import { HandleRegistry, type JobRecord } from '../tools/handles.js'; +import { Injector } from './injector.js'; + +const DEFAULT_GRACEFUL_STOP_DRAIN_MS = 30_000; +const MAX_ACCESSIBILITY_CHARS = 8_000; +const MAX_VOICE_CONTEXT_ENTRIES = 12; +const MAX_VOICE_CONTEXT_CHARS = 4_000; +const MAX_SPOKEN_SUMMARY_CHARS = 200; + +/** + * The Host surface LiveSession drives. Structurally satisfied by the ported + * LiveHostCoordinator. + */ +export interface LiveHostControl { + setCallState( + epoch: number, + state: Exclude, + ): boolean; + /** Registers the live session as the appshot-authorized caller. */ + setCoordinator( + epoch: number, + locator: { workspaceCwd: string; sessionId: string }, + ): boolean; + sendOutputAudio(epoch: number, pcm16: Uint8Array): boolean; + clearOutput(epoch: number): void; + setCaption(epoch: number, caption: string): boolean; + setStatusText(epoch: number, statusText?: string): boolean; + setTranscript?(epoch: number, transcript: string): boolean; + failCall(epoch: number, message?: string): boolean; + captureScreenContext( + callerSessionId: string, + ): Promise; +} + +export interface LiveRealtimeConfig { + endpoint: string; + apiKey?: string; + model: string; + voice?: string; +} + +export interface LiveSessionOptions { + host: LiveHostControl; + adaptor: BackendAdaptor; + realtime: LiveRealtimeConfig; + log: SessionLog; + openRealtime?: typeof openQwenRealtimeSession; + gracefulStopDrainMs?: number; +} + +interface CallContext { + epoch: number; + callId: string; + realtime?: QwenRealtimeSession; + stopping: boolean; + speechInProgress: boolean; + responseInFlight: boolean; + caption: string; + defaultSessionHandle?: string; + /** Per backend-session event pump cancellation. */ + pumps: Map; + injector: Injector; + broker: PermissionBroker; + stopResolve?: (outcome: void | { error: string }) => void; +} + +/** + * Sentence boundaries for spoken clamps. ASCII terminators count only when + * followed by whitespace or end-of-string (a period inside a file path, IP, + * or version must not end a "sentence"); CJK terminators (。！？) count + * unconditionally — standard CJK typography puts no space after them. + */ +const SENTENCE_BOUNDARY = /(?<=[.!?])\s+|(?<=[。！？])\s*/; + +function splitSentences(text: string): string[] { + return text + .split(SENTENCE_BOUNDARY) + .map((part) => part.trim()) + .filter(Boolean); +} + +function firstSentence(text: string, max: number): string { + const trimmed = text.trim().replace(/\s+/g, ' '); + if (!trimmed) return ''; + const sentence = splitSentences(trimmed)[0] ?? trimmed; + return sentence.length > max ? `${sentence.slice(0, max)}…` : sentence; +} + +/** + * The spoken take-away from a long result: its closing sentence. Backend + * summaries are tail-clamped, so the head may start mid-sentence — the + * final sentence is the model's own conclusion and always complete. + */ +function lastSentence(text: string, max: number): string { + const trimmed = text.trim().replace(/\s+/g, ' '); + if (!trimmed) return ''; + const parts = splitSentences(trimmed); + const sentence = parts[parts.length - 1] ?? trimmed; + return sentence.length > max ? `${sentence.slice(0, max)}…` : sentence; +} + +function formatVoiceContext( + entries: readonly RealtimeTranscriptEntry[], +): string { + const recent = entries.slice(-MAX_VOICE_CONTEXT_ENTRIES); + let block = recent + .map( + (entry) => + `${entry.role === 'user' ? 'User' : 'Assistant'}: ${entry.text}`, + ) + .join('\n'); + if (block.length > MAX_VOICE_CONTEXT_CHARS) { + block = `…${block.slice(block.length - MAX_VOICE_CONTEXT_CHARS)}`; + } + return block; +} + +export class LiveSession { + private readonly host: LiveHostControl; + private readonly adaptor: BackendAdaptor; + private readonly log: SessionLog; + private readonly openRealtime: typeof openQwenRealtimeSession; + private readonly gracefulStopDrainMs: number; + private readonly handles = new HandleRegistry(); + private active?: CallContext; + + constructor(private readonly options: LiveSessionOptions) { + this.host = options.host; + this.adaptor = options.adaptor; + this.log = options.log; + this.openRealtime = options.openRealtime ?? openQwenRealtimeSession; + this.gracefulStopDrainMs = + options.gracefulStopDrainMs ?? DEFAULT_GRACEFUL_STOP_DRAIN_MS; + } + + /** LiveCallHandlers.onStart */ + async start(call: { + epoch: number; + callId: string; + mode: 'resume' | 'new'; + }): Promise { + this.closeActive(); + const context: CallContext = { + epoch: call.epoch, + callId: call.callId, + stopping: false, + speechInProgress: false, + responseInFlight: false, + caption: '', + pumps: new Map(), + injector: new Injector({ + sink: { + injectContext: (text) => this.injectContext(context, text), + injectSpeech: (text) => this.injectSpeech(context, text), + onInjected: (item, spoken) => { + this.log.write(spoken ? 'inject.speech' : 'inject.context', { + kind: item.kind, + job: item.jobHandle, + chars: item.context.length, + }); + }, + }, + }), + broker: new PermissionBroker({ + adaptor: this.adaptor, + log: (type, payload) => this.log.write(type, payload), + }), + }; + this.active = context; + this.log.write('session.start', { + callId: call.callId, + epoch: call.epoch, + mode: call.mode, + adaptor: this.adaptor.name, + model: this.options.realtime.model, + voice: this.options.realtime.voice, + }); + this.host.setCallState(call.epoch, 'starting'); + // Register the live call itself as the appshot-authorized caller; the + // ported host coordinator gates screen capture on this locator. + this.host.setCoordinator(call.epoch, { + workspaceCwd: '/', + sessionId: call.callId, + }); + + try { + const realtime = await this.openRealtime( + { + endpoint: this.options.realtime.endpoint, + ...(this.options.realtime.apiKey + ? { apiKey: this.options.realtime.apiKey } + : {}), + model: this.options.realtime.model, + callEpoch: call.epoch, + ...(this.options.realtime.voice + ? { voice: this.options.realtime.voice } + : {}), + instructions: buildLiveInstructions(), + tools: LIVE_SESSION_TOOLS, + }, + this.callbacksFor(context), + ); + if (this.active !== context || context.stopping) { + realtime.close({ discardPendingInput: true }); + return; + } + context.realtime = realtime; + this.host.setCallState(call.epoch, 'listening'); + } catch (error) { + this.log.write('error', { + source: 'realtime', + message: error instanceof Error ? error.message : String(error), + }); + if (this.active === context) { + this.host.failCall(call.epoch, 'Live Voice could not connect.'); + this.active = undefined; + } + throw error; + } + } + + /** LiveCallHandlers.onStop */ + stop(call: { + epoch: number; + callId: string; + }): Promise { + const context = this.active; + if (!context || context.epoch !== call.epoch) return Promise.resolve(); + if (context.stopping) { + return new Promise((resolve) => { + const previous = context.stopResolve; + context.stopResolve = (outcome) => { + previous?.(outcome); + resolve(outcome); + }; + }); + } + context.stopping = true; + this.host.clearOutput(context.epoch); + this.host.setCallState(context.epoch, 'stopping'); + + return new Promise((resolve) => { + context.stopResolve = resolve; + const finish = (outcome: void | { error: string }) => { + if (this.active === context) this.finishStop(context, outcome); + }; + // Commit any trailing speech so the provider transcribes it, then wait + // for the in-flight response to settle — bounded by the drain budget. + // The commit ack (onInputCommitted) clears speechInProgress, so the + // drain settles deterministically instead of burning the full budget. + if (context.speechInProgress) { + let committed = false; + try { + committed = context.realtime?.commitInputAudio() ?? false; + } catch { + committed = false; + } + if (!committed) { + finish({ + error: 'Live Voice could not commit the final spoken input.', + }); + return; + } + } + if (!context.responseInFlight && !context.speechInProgress) { + finish(undefined); + return; + } + const timer = setTimeout(() => { + finish({ + error: + 'Live Voice could not confirm the final spoken input before the stop deadline.', + }); + }, this.gracefulStopDrainMs); + timer.unref?.(); + context.injector.dispose(); + const poll = setInterval(() => { + if (this.active !== context) { + clearInterval(poll); + clearTimeout(timer); + return; + } + if (!context.responseInFlight && !context.speechInProgress) { + clearInterval(poll); + clearTimeout(timer); + finish(undefined); + } + }, 100); + poll.unref?.(); + }); + } + + /** LiveCallHandlers.onInputAudio */ + pushAudio(call: { epoch: number; callId: string; pcm16: Buffer }): boolean { + const context = this.active; + if (!context || context.epoch !== call.epoch || context.stopping) { + return true; // stale frames are dropped, not fatal + } + if (!context.realtime) return true; // still connecting + try { + // Propagate the provider's backpressure signal: a false return + // means the socket buffer is over its cap and frames are being + // dropped — the port source fails the call rather than letting VAD + // and transcription run on a gappy utterance. + return context.realtime.pushAudio(call.pcm16); + } catch { + return false; + } + } + + dispose(): void { + this.closeActive(); + } + + // -- realtime callbacks --------------------------------------------------- + + private callbacksFor(context: CallContext) { + const current = (session?: QwenRealtimeSession): boolean => + this.active === context && + (context.realtime === undefined || session === undefined + ? true + : context.realtime === session); + + return { + onReady: () => { + if (!current()) return; + this.log.write('session.start', { phase: 'realtime_ready' }); + }, + onSpeechStarted: () => { + if (!current()) return; + context.speechInProgress = true; + context.injector.noteSpeechStarted(); + this.log.write('vad.speech_started', {}); + }, + onSpeechStopped: () => { + if (!current()) return; + context.speechInProgress = false; + context.injector.noteSpeechStopped(); + this.log.write('vad.speech_stopped', {}); + }, + // The provider's input-commit ack: the utterance is out of the buffer, + // so speech is no longer "in progress" for the stop drain / injector. + // Once stopping, pushAudio drops frames, so this ack (or the transcript + // final below) is the only remaining clearer. + onInputCommitted: () => { + if (!current()) return; + context.speechInProgress = false; + this.log.write('vad.speech_stopped', { phase: 'input_committed' }); + }, + onInputTranscriptDone: (event: { text: string }) => { + if (!current()) return; + context.speechInProgress = false; + this.host.setTranscript?.(context.epoch, event.text); + this.log.write('transcript.user', { text: event.text }); + }, + onOutputTextDelta: (event: { text: string; source: string }) => { + if (!current()) return; + context.caption = `${context.caption}${event.text}`; + this.host.setCaption(context.epoch, context.caption); + }, + onOutputTextDone: (event: { text: string }) => { + if (!current()) return; + context.caption = ''; + this.log.write('transcript.assistant', { text: event.text }); + }, + onOutputAudioDelta: (event: { audio: Uint8Array }) => { + if (!current()) return; + this.host.sendOutputAudio(context.epoch, event.audio); + context.injector.noteOutputAudio(event.audio.byteLength); + }, + onResponseCreated: (event: { responseId: string; authority: string }) => { + if (!current()) return; + context.responseInFlight = true; + context.injector.noteResponseCreated(); + // During the stop drain the call state must stay 'stopping' — a + // 'speaking' flip here would strand the coordinator (its finish/fail + // paths early-return unless the call is still 'stopping'). + if (!context.stopping) { + this.host.setCallState(context.epoch, 'speaking'); + } + this.log.write('response.created', { + responseId: event.responseId, + authority: event.authority, + }); + }, + onResponseDone: (event: { responseId: string }) => { + if (!current()) return; + context.responseInFlight = false; + context.caption = ''; + context.injector.noteResponseDone(); + if (!context.stopping) { + this.host.setCallState(context.epoch, 'listening'); + } + this.log.write('response.done', { responseId: event.responseId }); + }, + onBargeIn: (event: { responseId: string }) => { + if (!current()) return; + this.host.clearOutput(context.epoch); + this.host.setCaption(context.epoch, ''); + this.host.setStatusText(context.epoch); + context.injector.noteOutputCleared(); + this.log.write('playback.cleared', { + reason: 'barge_in', + responseId: event.responseId, + }); + }, + onFunctionCall: (event: RealtimeFunctionCall) => { + if (!current()) return; + void this.dispatchTool(context, event); + }, + onDirectTranscript: (event: { + entries: readonly RealtimeTranscriptEntry[]; + }) => { + if (!current()) return; + for (const entry of event.entries) { + this.log.write( + entry.role === 'user' ? 'transcript.user' : 'transcript.assistant', + { text: entry.text, direct: true }, + ); + } + }, + onAudioDropped: () => { + // The provider is dropping mic frames (socket buffer over its + // cap): speech would run on a gappy utterance with no error + // surfaced — fail the call instead, mirroring the port source. + if (this.active !== context) return; + this.log.write('error', { + source: 'realtime', + message: 'audio frames were dropped: provider socket backpressured', + }); + this.host.failCall(context.epoch, 'audio frames were dropped'); + }, + onError: (error: { message: string; fatal: boolean; code?: string }) => { + if (!current()) return; + this.log.write('error', { + source: 'realtime', + code: error.code, + message: error.message, + fatal: error.fatal, + }); + if (error.fatal && context.stopping) { + // The socket is done for; the stop drain would otherwise wait the + // full budget for response/speech flags that can never settle. + context.responseInFlight = false; + context.speechInProgress = false; + } + if (error.fatal && !context.stopping) { + this.host.failCall(context.epoch, 'Live Voice failed.'); + this.cleanupContext(context); + } + }, + onClose: (info: { reason: string }) => { + if (this.active !== context) return; + this.log.write('session.end', { reason: info.reason }); + if (context.stopping) { + context.responseInFlight = false; + context.speechInProgress = false; + return; + } + if (info.reason !== 'client') { + this.host.failCall(context.epoch, 'Live Voice disconnected.'); + this.cleanupContext(context); + } + }, + }; + } + + // -- tools ---------------------------------------------------------------- + + private async dispatchTool( + context: CallContext, + event: RealtimeFunctionCall, + ): Promise { + const dispatcher = new ToolDispatcher({ + handlers: this.toolHandlers(context), + }); + if (!context.stopping) { + this.host.setCallState(context.epoch, 'thinking'); + } + this.log.write('tool.call', { + name: event.name, + callId: event.callId, + args: event.arguments.slice(0, 2_000), + }); + const ctx: ToolContext = { activeTranscript: event.activeTranscript }; + const result = await dispatcher.dispatch(event.name, event.arguments, ctx); + // The realtime session rejects empty or oversized outputs; a stranded + // call would hang that response's arbitration. Clamp defensively. + let receipt = result.receipt; + if (receipt.length > QWEN_REALTIME_LIMITS.maxFunctionOutputChars) { + receipt = JSON.stringify({ + status: 'error', + note: 'The result was too large to return; check the session on screen.', + }); + } + if (!receipt.trim()) receipt = '{}'; + this.log.write('tool.result', { + name: event.name, + callId: event.callId, + ok: result.ok, + receipt: receipt.slice(0, 2_000), + }); + if (this.active !== context || !context.realtime) return; + try { + context.realtime.submitFunctionOutput( + { callEpoch: context.epoch, callId: event.callId }, + receipt, + ); + } catch (error) { + this.log.write('error', { + source: 'tool_output', + message: error instanceof Error ? error.message : String(error), + }); + } + } + + private toolHandlers(context: CallContext): ReadonlyMap { + const handlers = new Map(); + + handlers.set(APPSHOT_TOOL_NAME, async () => { + const capture = await this.host.captureScreenContext(context.callId); + const asset = this.handles.registerAsset({ + path: capture.screenshotPath, + mimeType: 'image/png', + }); + return { + status: 'ok', + app: capture.appName, + ...(capture.windowTitle ? { window: capture.windowTitle } : {}), + accessibility_text: capture.accessibilityText.slice( + 0, + MAX_ACCESSIBILITY_CHARS, + ), + asset: asset.assetHandle, + }; + }); + + handlers.set(SESSION_LIST_TOOL_NAME, async () => { + const sessions = await this.adaptor.listSessions(); + return { + status: 'ok', + sessions: sessions.map((summary) => { + const handle = this.handles.session(summary.handle); + // Reconcile stale non-terminal jobs: a turn_complete emitted + // while no pump was subscribed (pumps are per-call and aborted + // at call end) would otherwise keep session_list reporting a + // running active_job forever. Gated on the backend's own idle + // report so a genuinely busy session is never touched. + if ( + summary.state !== 'busy' && + !this.adaptor.isBusy(summary.handle) + ) { + this.handles.reconcileIdleSession(handle); + } + const activeJob = this.handles.activeJobForSession(handle); + return { + handle, + ...(summary.label ? { label: summary.label } : {}), + ...(summary.cwd ? { cwd: summary.cwd } : {}), + state: this.adaptor.isBusy(summary.handle) ? 'busy' : summary.state, + ...(activeJob ? { active_job: activeJob.jobHandle } : {}), + }; + }), + }; + }); + + handlers.set(SESSION_CREATE_TOOL_NAME, async (args) => { + const backend = await this.adaptor.createSession({ + ...(typeof args['cwd'] === 'string' ? { cwd: args['cwd'] } : {}), + ...(typeof args['label'] === 'string' ? { label: args['label'] } : {}), + }); + const handle = this.handles.session(backend); + this.ensurePump(context, handle, backend); + return { status: 'ok', handle }; + }); + + handlers.set(HANDOFF_TOOL_NAME, async (args, ctx) => { + const task = typeof args['task'] === 'string' ? args['task'].trim() : ''; + if (!task) { + return { status: 'error', note: 'handoff needs a task.' }; + } + const target = await this.resolveHandoffTarget(context, args['session']); + if ('error' in target) return { status: 'error', note: target.error }; + const { handle, backend } = target; + + const blocks = await this.buildHandoffBlocks( + task, + ctx.activeTranscript, + args['input_refs'], + ); + const busy = this.adaptor.isBusy(backend); + const receipt = await this.adaptor.prompt(backend, blocks, { + steer: busy, + }); + if (receipt.status === 'rejected') { + return { + status: 'rejected', + session: handle, + note: receipt.note ?? 'the session refused the task', + }; + } + // A steer that joined the running turn comes back with that turn's + // jobRef: the instruction became part of the EXISTING job. Creating a + // second record would orphan the first in 'running' forever (nothing + // would ever transition it out). + const existing = + receipt.jobRef !== undefined + ? this.handles.jobByRef(receipt.jobRef) + : undefined; + const job = + existing ?? + this.handles.createJob({ + sessionHandle: handle, + backend, + ...(receipt.jobRef !== undefined ? { jobRef: receipt.jobRef } : {}), + task, + }); + this.ensurePump(context, handle, backend); + return { + status: receipt.status, + job: job.jobHandle, + session: handle, + ...(receipt.note ? { note: receipt.note } : {}), + }; + }); + + handlers.set(SESSION_MONITOR_TOOL_NAME, (args) => { + const job = + typeof args['job'] === 'string' + ? this.handles.resolveJob(args['job']) + : undefined; + const sessionHandle = + job?.sessionHandle ?? + (typeof args['session'] === 'string' ? args['session'].trim() : ''); + const backend = this.handles.resolveSession(sessionHandle); + if (!backend) { + return { + status: 'error', + note: 'unknown session; call session_list first.', + }; + } + if (!this.adaptor.isBusy(backend)) { + this.handles.reconcileIdleSession(sessionHandle); + } + const activeJob = job ?? this.handles.activeJobForSession(sessionHandle); + return { + status: 'ok', + session: sessionHandle, + state: this.adaptor.isBusy(backend) ? 'busy' : 'idle', + ...(activeJob + ? { + job: activeJob.jobHandle, + job_state: activeJob.state, + task: activeJob.task.slice(0, 200), + } + : {}), + }; + }); + + handlers.set(SESSION_STOP_TOOL_NAME, async (args) => { + const job = + typeof args['job'] === 'string' + ? this.handles.resolveJob(args['job']) + : undefined; + const sessionHandle = + job?.sessionHandle ?? + (typeof args['session'] === 'string' ? args['session'].trim() : ''); + const backend = + job?.backend ?? this.handles.resolveSession(sessionHandle); + if (!backend) { + return { + status: 'error', + note: 'unknown session or job; call session_list first.', + }; + } + await this.adaptor.cancel(backend); + if (job) job.state = 'cancelled'; + return { status: 'cancelling', session: sessionHandle }; + }); + + handlers.set(RESPOND_PERMISSION_TOOL_NAME, async (args) => { + const requestHandle = + typeof args['request_id'] === 'string' ? args['request_id'] : ''; + const decision = args['decision']; + if ( + decision !== 'allow' && + decision !== 'allow_always' && + decision !== 'deny' + ) { + return { + status: 'error', + note: 'decision must be allow, allow_always, or deny.', + }; + } + const note = typeof args['note'] === 'string' ? args['note'].trim() : ''; + // Resolve before respond(): a delivered vote clears the pending entry, + // and the backend handle is needed to relay the user's constraint. + const pending = context.broker.resolveHandle(requestHandle); + const outcome = await context.broker.respond( + requestHandle, + decision, + note || undefined, + ); + if (outcome === 'not_found') { + return { + status: 'error', + note: `no pending request ${requestHandle}.`, + }; + } + // The vote channel carries no free text; a user constraint ("only this + // file") would otherwise be silently discarded — the grant would be + // broader than the user believes. Relay it as a user instruction to + // the same backend session through the existing prompt/steer path. + if (note && pending && outcome === 'delivered') { + try { + await this.adaptor.prompt( + pending.backend, + [ + { + type: 'text', + text: + `The user answered the permission request "${pending.title}" ` + + `with "${decision}" and added this constraint, which you must ` + + `follow: ${note}`, + }, + ], + { steer: this.adaptor.isBusy(pending.backend) }, + ); + } catch (error) { + this.log.write('error', { + source: 'permission', + message: error instanceof Error ? error.message : String(error), + }); + return { + status: outcome, + note: + 'The vote was delivered, but the added constraint could not ' + + 'be relayed to the session; tell the user to check it on screen.', + }; + } + } + return { status: outcome }; + }); + + return handlers; + } + + private async resolveHandoffTarget( + context: CallContext, + sessionArg: unknown, + ): Promise<{ handle: string; backend: BackendHandle } | { error: string }> { + if (typeof sessionArg === 'string' && sessionArg.trim()) { + const backend = this.handles.resolveSession(sessionArg); + if (!backend) { + return { error: `unknown session ${sessionArg}; call session_list.` }; + } + return { handle: sessionArg.trim(), backend }; + } + if (context.defaultSessionHandle) { + const backend = this.handles.resolveSession(context.defaultSessionHandle); + if (backend) { + return { handle: context.defaultSessionHandle, backend }; + } + } + const backend = await this.adaptor.createSession({ label: 'Voice chat' }); + const handle = this.handles.session(backend); + context.defaultSessionHandle = handle; + return { handle, backend }; + } + + private async buildHandoffBlocks( + task: string, + activeTranscript: readonly RealtimeTranscriptEntry[], + inputRefs: unknown, + ): Promise { + const parts = [task]; + const voiceContext = formatVoiceContext(activeTranscript); + if (voiceContext) { + parts.push( + `\nRelayed from the user's live voice conversation; use it only to resolve references in the task.\n${voiceContext}\n`, + ); + } + const blocks: ContentBlock[] = [{ type: 'text', text: parts.join('\n\n') }]; + if (Array.isArray(inputRefs)) { + for (const ref of inputRefs) { + if (typeof ref !== 'string') continue; + const asset = this.handles.resolveAsset(ref); + if (!asset) continue; + try { + const data = await readFile(asset.path); + blocks.push({ + type: 'image', + mimeType: asset.mimeType, + data: new Uint8Array(data), + name: `${asset.assetHandle}.png`, + }); + } catch { + /* the capture expired; the text task still stands */ + } + } + } + return blocks; + } + + // -- backend event pump --------------------------------------------------- + + private ensurePump( + context: CallContext, + sessionHandle: string, + backend: BackendHandle, + ): void { + if (context.pumps.has(sessionHandle)) return; + const abort = new AbortController(); + context.pumps.set(sessionHandle, abort); + void this.pump(context, sessionHandle, backend, abort.signal).catch( + (error) => { + this.log.write('error', { + source: 'pump', + session: sessionHandle, + message: error instanceof Error ? error.message : String(error), + }); + }, + ); + } + + private async pump( + context: CallContext, + sessionHandle: string, + backend: BackendHandle, + signal: AbortSignal, + ): Promise { + // The SSE stream can end without a session_closed (daemon restart, + // dropped connection). Resubscribe with backoff instead of leaving the + // session permanently unobserved — completion events would be lost. + let backoffMs = 1_000; + while (this.active === context && !signal.aborted) { + let sawEvent = false; + try { + for await (const event of this.adaptor.events(backend, { signal })) { + if (this.active !== context) return; + sawEvent = true; + backoffMs = 1_000; + this.log.write('backend.event', { + session: sessionHandle, + type: event.type, + ...('jobRef' in event && event.jobRef !== undefined + ? { jobRef: event.jobRef } + : {}), + }); + this.onBackendEvent(context, sessionHandle, backend, event); + if (event.type === 'session_closed') { + context.pumps.delete(sessionHandle); + return; + } + } + } catch (error) { + if (signal.aborted || this.active !== context) break; + this.log.write('error', { + source: 'pump', + session: sessionHandle, + message: error instanceof Error ? error.message : String(error), + }); + } + if (signal.aborted || this.active !== context) break; + this.log.write('backend.event', { + session: sessionHandle, + type: 'stream_ended', + resubscribeInMs: backoffMs, + sawEvent, + }); + await new Promise((resolve) => { + const timer = setTimeout(resolve, backoffMs); + timer.unref?.(); + }); + backoffMs = Math.min(backoffMs * 2, 10_000); + } + context.pumps.delete(sessionHandle); + } + + private onBackendEvent( + context: CallContext, + sessionHandle: string, + backend: BackendHandle, + event: BackendEvent, + ): void { + switch (event.type) { + case 'turn_started': { + const job = event.jobRef + ? this.handles.jobByRef(event.jobRef) + : undefined; + if (job) job.state = 'running'; + return; + } + case 'progress': { + const job = this.jobFor(sessionHandle, event.jobRef); + context.injector.enqueue({ + kind: 'progress', + context: `[PROGRESS ${job?.jobHandle ?? sessionHandle}] ${event.summary}`, + ...(job ? { jobHandle: job.jobHandle } : {}), + }); + return; + } + case 'speak': { + context.injector.enqueue({ + kind: 'speak', + context: `[BACKEND ${sessionHandle}] ${event.text}`, + spoken: event.text.slice(0, MAX_SPOKEN_SUMMARY_CHARS), + }); + return; + } + case 'turn_complete': { + const job = this.jobFor(sessionHandle, event.jobRef); + if (job) job.state = 'done'; + const label = job?.jobHandle ?? sessionHandle; + const spokenSummary = lastSentence( + event.summary, + MAX_SPOKEN_SUMMARY_CHARS, + ); + context.injector.enqueue({ + kind: 'complete', + context: `[COMPLETE ${label}] ${event.detail ?? event.summary}`, + spoken: spokenSummary + ? `Task ${label} finished. ${spokenSummary}` + : `Task ${label} finished.`, + ...(job ? { jobHandle: job.jobHandle } : {}), + }); + return; + } + case 'turn_error': { + const job = this.jobFor(sessionHandle, event.jobRef); + if (job) + job.state = event.error === 'cancelled' ? 'cancelled' : 'failed'; + const label = job?.jobHandle ?? sessionHandle; + if (event.error === 'cancelled') { + context.injector.enqueue({ + kind: 'complete', + context: `[COMPLETE ${label}] cancelled at the user's request.`, + }); + return; + } + context.injector.enqueue({ + kind: 'error', + context: `[ERROR ${label}] ${event.error}`, + spoken: `Task ${label} hit a problem. ${firstSentence(event.error, 120)}`, + ...(job ? { jobHandle: job.jobHandle } : {}), + }); + return; + } + case 'permission_request': { + void context.broker + .onRequest({ + requestId: event.requestId, + backend, + sessionHandle, + title: event.title, + options: event.options, + }) + .then((ask) => { + if (ask.autoAnswered || this.active !== context) return; + context.injector.enqueue({ + kind: 'permission', + requestId: event.requestId, + context: `[PERMISSION ${ask.pending.requestHandle}] Session ${sessionHandle} wants to run: ${event.title}. Ask the user and relay their answer with respond_permission.`, + spoken: `The task wants to ${event.title}. Should I allow it?`, + }); + }) + .catch((error: unknown) => { + // A rejected broker chain must never become an unhandled + // rejection (it would take the whole daemon down mid-call). + this.log.write('error', { + source: 'permission', + message: error instanceof Error ? error.message : String(error), + }); + if (this.active === context) { + context.injector.enqueue({ + kind: 'error', + context: `[ERROR ${sessionHandle}] A permission request could not be processed; the task may be stuck waiting for approval.`, + spoken: + 'A task is waiting for an approval I could not process. You may need to check it on screen.', + }); + } + }); + return; + } + case 'permission_resolved': { + const pending = context.broker.onResolved(event.requestId); + if (!event.byUs) { + const retracted = context.injector.retractPermission(event.requestId); + if (!retracted && pending) { + context.injector.enqueue({ + kind: 'progress', + context: `[BACKEND ${sessionHandle}] The permission request (${pending.requestHandle}) was already handled elsewhere; no answer needed.`, + }); + } + } + return; + } + case 'session_closed': { + // A closed session must not keep resolving every session-less + // handoff to a dead target for the rest of the call (WebShell + // deletion, daemon restart, idle reaper): stop resolving its + // handle entirely and clear the default so + // resolveHandoffTarget's createSession fall-through rebuilds. + this.handles.closeSession(sessionHandle); + if (context.defaultSessionHandle === sessionHandle) { + context.defaultSessionHandle = undefined; + } + return; + } + default: + return; + } + } + + private jobFor( + sessionHandle: string, + jobRef: string | undefined, + ): JobRecord | undefined { + if (jobRef) { + const byRef = this.handles.jobByRef(jobRef); + if (byRef) return byRef; + } + return this.handles.activeJobForSession(sessionHandle); + } + + // -- injection sinks ------------------------------------------------------- + + private injectContext(context: CallContext, text: string): boolean { + if (this.active !== context || !context.realtime || context.stopping) { + return false; + } + try { + return context.realtime.sendBackendContext(text); + } catch { + return false; + } + } + + private injectSpeech(context: CallContext, text: string): boolean { + if (this.active !== context || !context.realtime || context.stopping) { + return false; + } + try { + return context.realtime.speakToUser(text); + } catch { + return false; + } + } + + // -- teardown --------------------------------------------------------------- + + private finishStop( + context: CallContext, + outcome: void | { error: string }, + ): void { + const resolve = context.stopResolve; + context.stopResolve = undefined; + this.cleanupContext(context); + this.log.write('session.end', { + callId: context.callId, + ...(outcome && 'error' in outcome ? { error: outcome.error } : {}), + }); + resolve?.(outcome); + } + + private cleanupContext(context: CallContext): void { + if (this.active === context) this.active = undefined; + context.injector.dispose(); + for (const abort of context.pumps.values()) abort.abort(); + context.pumps.clear(); + try { + context.realtime?.close({ discardPendingInput: true }); + } catch { + /* already closed */ + } + // A stop() waiter must never be left hanging when the context is torn + // down through another path (daemon shutdown, fatal error). + const resolve = context.stopResolve; + context.stopResolve = undefined; + resolve?.(undefined); + } + + private closeActive(): void { + const context = this.active; + if (!context) return; + this.cleanupContext(context); + } +} diff --git a/packages/qwen-live/src/permissions/permission-broker.test.ts b/packages/qwen-live/src/permissions/permission-broker.test.ts new file mode 100644 index 00000000000..470fcda967b --- /dev/null +++ b/packages/qwen-live/src/permissions/permission-broker.test.ts @@ -0,0 +1,378 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it, vi } from 'vitest'; +import type { + BackendAdaptor, + BackendHandle, + PermissionOption, +} from '../adaptor/types.js'; +import { PermissionBroker } from './permission-broker.js'; + +const BACKEND: BackendHandle = { id: 's1', adaptor: 'fake' }; + +const OPTIONS: readonly PermissionOption[] = [ + { optionId: 'allow', kind: 'proceed' }, + { optionId: 'deny', kind: 'reject' }, +]; + +function createAdaptor() { + return { + respondPermission: vi.fn( + async (): Promise<'delivered' | 'already_resolved'> => 'delivered', + ), + }; +} + +type FakeAdaptor = ReturnType; + +interface Rig { + adaptor: FakeAdaptor; + broker: PermissionBroker; + clock: { now: number }; + logEvents: Array<{ type: string; payload: Record }>; +} + +function createBroker(options?: { ruleTtlMs?: number }): Rig { + const adaptor = createAdaptor(); + const clock = { now: 1_000_000 }; + const logEvents: Rig['logEvents'] = []; + const broker = new PermissionBroker({ + adaptor: adaptor as unknown as BackendAdaptor, + now: () => clock.now, + ...(options?.ruleTtlMs !== undefined + ? { ruleTtlMs: options.ruleTtlMs } + : {}), + log: (type, payload) => { + logEvents.push({ type, payload }); + }, + }); + return { adaptor, broker, clock, logEvents }; +} + +function request( + broker: PermissionBroker, + fields?: { requestId?: string; sessionHandle?: string; title?: string }, +) { + return broker.onRequest({ + requestId: fields?.requestId ?? 'r1', + backend: BACKEND, + sessionHandle: fields?.sessionHandle ?? 'session_1', + title: fields?.title ?? 'Bash: rm -rf /a', + options: OPTIONS, + }); +} + +describe('PermissionBroker', () => { + it('records pending asks with incrementing handles when no rule matches', async () => { + const { adaptor, broker } = createBroker(); + + const first = await request(broker, { requestId: 'r1' }); + expect(first.autoAnswered).toBe(false); + expect(first.pending.requestHandle).toBe('req_1'); + expect(first.pending.requestId).toBe('r1'); + expect(first.pending.sessionHandle).toBe('session_1'); + + const second = await request(broker, { requestId: 'r2' }); + expect(second.autoAnswered).toBe(false); + expect(second.pending.requestHandle).toBe('req_2'); + + expect(broker.pendingCount).toBe(2); + expect(adaptor.respondPermission).not.toHaveBeenCalled(); + }); + + it('delivers an allow vote and clears the pending ask', async () => { + const { adaptor, broker } = createBroker(); + await request(broker, { requestId: 'r1' }); + + const outcome = await broker.respond('req_1', 'allow'); + + expect(outcome).toBe('delivered'); + expect(adaptor.respondPermission).toHaveBeenCalledWith( + BACKEND, + 'r1', + 'allow', + ); + expect(broker.pendingCount).toBe(0); + expect(broker.resolveHandle('req_1')).toBeUndefined(); + }); + + it('translates allow_always to a one-shot allow and auto-answers identical repeats', async () => { + const { adaptor, broker } = createBroker(); + await request(broker, { requestId: 'r1', title: 'Bash: rm -rf /a' }); + + await broker.respond('req_1', 'allow_always'); + // The protocol vote never carries the standing grant. + expect(adaptor.respondPermission).toHaveBeenLastCalledWith( + BACKEND, + 'r1', + 'allow', + ); + + // Same session, same normalized title: the grant covers the repeat. + const ask = await request(broker, { + requestId: 'r2', + title: 'Bash: rm -rf /a ', + }); + expect(ask.autoAnswered).toBe(true); + expect(adaptor.respondPermission).toHaveBeenLastCalledWith( + BACKEND, + 'r2', + 'allow', + ); + expect(broker.pendingCount).toBe(0); + }); + + it('scopes the standing rule to the whole approved command, not its prefix', async () => { + const { broker } = createBroker(); + await request(broker, { + requestId: 'r1', + title: 'Bash: git push origin main', + }); + await broker.respond('req_1', 'allow_always'); + await request(broker, { requestId: 'r2', title: 'Bash: rm -rf /a' }); + await broker.respond('req_2', 'allow_always'); + + // A flag variant is a different command; the user approved a command, + // not its destructive variants. + const forced = await request(broker, { + requestId: 'r3', + title: 'Bash: git push --force origin main', + }); + expect(forced.autoAnswered).toBe(false); + + // Same two-token prefix, different target: likewise not covered. + const differentTarget = await request(broker, { + requestId: 'r4', + title: 'Bash: rm -rf /b', + }); + expect(differentTarget.autoAnswered).toBe(false); + + // The identical repeat is. + const repeat = await request(broker, { + requestId: 'r5', + title: 'Bash: git push origin main', + }); + expect(repeat.autoAnswered).toBe(true); + }); + + it('does not auto-answer requests with a different title key', async () => { + const { broker } = createBroker(); + await request(broker, { requestId: 'r1', title: 'Bash: rm -rf /a' }); + await broker.respond('req_1', 'allow_always'); + + const ask = await request(broker, { + requestId: 'r2', + title: 'Bash: ls -la', + }); + expect(ask.autoAnswered).toBe(false); + }); + + it('stops auto-answering once the standing rule expires', async () => { + const { broker, clock } = createBroker({ ruleTtlMs: 60_000 }); + await request(broker, { requestId: 'r1' }); + await broker.respond('req_1', 'allow_always'); + + clock.now += 59_999; + const beforeExpiry = await request(broker, { requestId: 'r2' }); + expect(beforeExpiry.autoAnswered).toBe(true); + + clock.now += 2; + const afterExpiry = await request(broker, { requestId: 'r3' }); + expect(afterExpiry.autoAnswered).toBe(false); + }); + + it('scopes standing rules to the session that granted them', async () => { + const { broker } = createBroker(); + await request(broker, { requestId: 'r1', sessionHandle: 'session_1' }); + await broker.respond('req_1', 'allow_always'); + + const otherSession = await request(broker, { + requestId: 'r2', + sessionHandle: 'session_2', + }); + expect(otherSession.autoAnswered).toBe(false); + }); + + it('delivers deny votes and reports unknown handles', async () => { + const { adaptor, broker } = createBroker(); + await request(broker, { requestId: 'r1' }); + + const denied = await broker.respond('req_1', 'deny'); + expect(denied).toBe('delivered'); + expect(adaptor.respondPermission).toHaveBeenCalledWith( + BACKEND, + 'r1', + 'deny', + ); + + const missing = await broker.respond('req_99', 'allow'); + expect(missing).toBe('not_found'); + expect(adaptor.respondPermission).toHaveBeenCalledTimes(1); + }); + + it('onResolved returns the pending ask once and clears it', async () => { + const { broker } = createBroker(); + await request(broker, { requestId: 'r1' }); + + const pending = broker.onResolved('r1'); + expect(pending?.requestHandle).toBe('req_1'); + expect(broker.pendingCount).toBe(0); + + expect(broker.onResolved('r1')).toBeUndefined(); + }); + + it('logs requests and decisions with the auto flag', async () => { + const { broker, logEvents } = createBroker(); + + await request(broker, { requestId: 'r1', title: 'Bash: rm -rf /a' }); + await broker.respond('req_1', 'allow_always'); + await request(broker, { requestId: 'r2', title: 'Bash: rm -rf /a' }); + + expect(logEvents.map((event) => event.type)).toEqual([ + 'permission.request', + 'permission.decision', + 'permission.request', + 'permission.decision', + ]); + expect(logEvents[0]?.payload).toMatchObject({ + requestHandle: 'req_1', + requestId: 'r1', + session: 'session_1', + title: 'Bash: rm -rf /a', + }); + expect(logEvents[1]?.payload).toMatchObject({ + requestHandle: 'req_1', + decision: 'allow', + auto: false, + outcome: 'delivered', + }); + expect(logEvents[3]?.payload).toMatchObject({ + requestHandle: 'req_2', + requestId: 'r2', + decision: 'allow', + auto: true, + }); + }); + + it("records the user's spoken constraint note in the decision log", async () => { + const { broker, logEvents } = createBroker(); + await request(broker, { requestId: 'r1' }); + + const outcome = await broker.respond('req_1', 'allow', 'only this file'); + + expect(outcome).toBe('delivered'); + const decision = logEvents.find( + (event) => event.type === 'permission.decision', + ); + expect(decision?.payload).toMatchObject({ + requestHandle: 'req_1', + decision: 'allow', + note: 'only this file', + }); + }); + + it('clears the pending ask when the backend reports it already resolved', async () => { + const { adaptor, broker } = createBroker(); + await request(broker, { requestId: 'r1' }); + adaptor.respondPermission.mockResolvedValueOnce('already_resolved'); + + const outcome = await broker.respond('req_1', 'allow'); + + expect(outcome).toBe('already_resolved'); + expect(broker.pendingCount).toBe(0); + expect(broker.resolveHandle('req_1')).toBeUndefined(); + // The handle is gone: a second relay reports not_found instead of + // re-voting on a settled request. + expect(await broker.respond('req_1', 'allow')).toBe('not_found'); + expect(adaptor.respondPermission).toHaveBeenCalledTimes(1); + }); + + it('falls back to a spoken ask when the silent auto-approval fails to deliver', async () => { + const { adaptor, broker, logEvents } = createBroker(); + const first = await request(broker, { requestId: 'r1' }); + await broker.respond(first.pending.requestHandle, 'allow_always'); + + // The standing rule matches, but the backend is unreachable: the ask + // must surface to the user instead of crashing the caller. + adaptor.respondPermission.mockRejectedValueOnce( + new Error('daemon unreachable'), + ); + const second = await request(broker, { requestId: 'r2' }); + + expect(second.autoAnswered).toBe(false); + expect(broker.resolveHandle(second.pending.requestHandle)).toBeDefined(); + const failure = logEvents.find( + (event) => + event.type === 'permission.decision' && + event.payload['outcome'] === 'delivery_failed', + ); + expect(failure?.payload).toMatchObject({ auto: true, decision: 'allow' }); + }); + + it('keeps the whole command after colons in the standing-rule key', async () => { + // `git commit -m "fix: bug"` — the command itself contains a colon; + // the key must not truncate at it (or a different -m message with the + // same prefix would silently match). + const { broker } = createBroker(); + await request(broker, { + requestId: 'r1', + title: 'Shell: git commit -m "fix: bug"', + }); + await broker.respond('req_1', 'allow_always'); + + const different = await request(broker, { + requestId: 'r2', + title: 'Shell: git commit -m "fix: something else entirely"', + }); + expect(different.autoAnswered).toBe(false); + + const identical = await request(broker, { + requestId: 'r3', + title: 'Shell: git commit -m "fix: bug"', + }); + expect(identical.autoAnswered).toBe(true); + }); + + it('does not case-fold the standing-rule key — /a and /A are different paths', async () => { + const { broker } = createBroker(); + await request(broker, { requestId: 'r1', title: 'Bash: rm -rf /a' }); + await broker.respond('req_1', 'allow_always'); + + const different = await request(broker, { + requestId: 'r2', + title: 'Bash: rm -rf /A', + }); + expect(different.autoAnswered).toBe(false); + }); + + it('an explicit deny revokes the standing rule that matched the request', async () => { + const { broker, adaptor } = createBroker(); + await request(broker, { requestId: 'r1', title: 'Bash: rm -rf /a' }); + await broker.respond('req_1', 'allow_always'); + + // The silent auto-answer fails and falls back to a spoken ask… + adaptor.respondPermission.mockImplementation(async () => { + throw new Error('daemon unreachable'); + }); + const fallback = await request(broker, { + requestId: 'r2', + title: 'Bash: rm -rf /a', + }); + expect(fallback.autoAnswered).toBe(false); + + // …and the user says deny. The rule must not survive the refusal. + adaptor.respondPermission.mockImplementation( + async () => 'delivered' as const, + ); + await broker.respond('req_2', 'deny'); + const again = await request(broker, { + requestId: 'r3', + title: 'Bash: rm -rf /a', + }); + expect(again.autoAnswered).toBe(false); + }); +}); diff --git a/packages/qwen-live/src/permissions/permission-broker.ts b/packages/qwen-live/src/permissions/permission-broker.ts new file mode 100644 index 00000000000..6c5989156d6 --- /dev/null +++ b/packages/qwen-live/src/permissions/permission-broker.ts @@ -0,0 +1,240 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Voice-side permission handling. + * + * Design points carried over from the split design doc (§7): + * - "allow always" never reaches the backend as a persistent grant. The + * protocol vote is always a one-shot allow; the standing rule lives here + * with a TTL and is applied by silently auto-answering similar requests. + * - A request resolved elsewhere (WebShell) retracts the queued spoken ask. + */ + +import type { + BackendAdaptor, + BackendHandle, + PermissionOption, +} from '../adaptor/types.js'; + +const DEFAULT_RULE_TTL_MS = 30 * 60_000; +const MAX_RULES = 64; + +export interface PendingPermission { + requestHandle: string; + requestId: string; + backend: BackendHandle; + sessionHandle: string; + title: string; + options: readonly PermissionOption[]; + createdAt: number; +} + +export interface PermissionBrokerOptions { + adaptor: BackendAdaptor; + now?: () => number; + ruleTtlMs?: number; + log?: ( + type: 'permission.request' | 'permission.decision', + payload: Record, + ) => void; +} + +export type SpokenPermissionDecision = 'allow' | 'allow_always' | 'deny'; + +export interface PermissionAskEvent { + pending: PendingPermission; + /** True when a standing rule answered it silently — nothing to speak. */ + autoAnswered: boolean; +} + +interface StandingRule { + sessionHandle: string; + titleKey: string; + expiresAt: number; +} + +/** + * Standing-rule key: tool name plus the WHOLE normalized detail (trimmed, + * whitespace-collapsed — but case-preserved: `/a` and `/A` are different + * directories on Linux, and commands routinely contain colons, so the + * split is on the FIRST colon only and the entire remainder is the + * detail). A voice "always allow" is a trust grant; it must cover only + * repeats of the exact command the user heard and approved — + * `git push origin main` never silently covers + * `git push --force origin main`, and `git commit -m "fix: bug"` never + * covers `git commit -m "fix: something else entirely"`. + */ +function titleKeyOf(title: string): string { + const separator = title.indexOf(':'); + const head = separator === -1 ? title : title.slice(0, separator); + const detail = separator === -1 ? '' : title.slice(separator + 1); + const normalized = detail.trim().split(/\s+/).join(' '); + return `${head.trim()}:${normalized}`; +} + +export class PermissionBroker { + private readonly pending = new Map(); + private readonly pendingByRequestId = new Map(); + private rules: StandingRule[] = []; + private seq = 0; + private readonly now: () => number; + private readonly ruleTtlMs: number; + + constructor(private readonly options: PermissionBrokerOptions) { + this.now = options.now ?? Date.now; + this.ruleTtlMs = options.ruleTtlMs ?? DEFAULT_RULE_TTL_MS; + } + + /** + * A permission_request arrived from the backend. Returns what the caller + * should do: speak the ask, or nothing (a standing rule auto-answered). + */ + async onRequest(fields: { + requestId: string; + backend: BackendHandle; + sessionHandle: string; + title: string; + options: readonly PermissionOption[]; + }): Promise { + const pending: PendingPermission = { + requestHandle: `req_${++this.seq}`, + requestId: fields.requestId, + backend: fields.backend, + sessionHandle: fields.sessionHandle, + title: fields.title, + options: fields.options, + createdAt: this.now(), + }; + this.pending.set(pending.requestHandle, pending); + this.pendingByRequestId.set(pending.requestId, pending.requestHandle); + this.options.log?.('permission.request', { + requestHandle: pending.requestHandle, + requestId: pending.requestId, + session: pending.sessionHandle, + title: pending.title, + }); + + if (this.matchesRule(pending)) { + // A failed silent delivery must not crash the caller or swallow the + // request: fall back to asking the user aloud. + try { + await this.deliver(pending, 'allow', true); + return { pending, autoAnswered: true }; + } catch (error) { + this.options.log?.('permission.decision', { + requestHandle: pending.requestHandle, + requestId: pending.requestId, + decision: 'allow', + auto: true, + outcome: 'delivery_failed', + error: error instanceof Error ? error.message : String(error), + }); + return { pending, autoAnswered: false }; + } + } + return { pending, autoAnswered: false }; + } + + /** + * The user answered aloud (via the respond_permission tool). `note` is a + * spoken constraint the user attached ("only this file"); the vote channel + * cannot carry it, so it is recorded in the decision log here and relayed + * to the backend session by the caller. + */ + async respond( + requestHandle: string, + decision: SpokenPermissionDecision, + note?: string, + ): Promise<'delivered' | 'already_resolved' | 'not_found'> { + const pending = this.pending.get(requestHandle.trim()); + if (!pending) return 'not_found'; + if (decision === 'allow_always') { + this.remember(pending); + } + if (decision === 'deny') { + // An explicit refusal must outrank any standing rule that matched + // this same request (the silent auto-answer failed and fell back to + // asking): without revocation the invisible rule silently overrides + // the user's spoken "no" on the next identical request. Broker-local + // per the design note — the protocol vote has no revocation concept. + const key = titleKeyOf(pending.title); + this.rules = this.rules.filter( + (rule) => + rule.sessionHandle !== pending.sessionHandle || rule.titleKey !== key, + ); + } + return await this.deliver( + pending, + decision === 'deny' ? 'deny' : 'allow', + false, + note, + ); + } + + /** A resolution arrived from the event stream (possibly our own vote). */ + onResolved(requestId: string): PendingPermission | undefined { + const handle = this.pendingByRequestId.get(requestId); + if (handle === undefined) return undefined; + this.pendingByRequestId.delete(requestId); + const pending = this.pending.get(handle); + this.pending.delete(handle); + return pending; + } + + resolveHandle(requestHandle: string): PendingPermission | undefined { + return this.pending.get(requestHandle.trim()); + } + + get pendingCount(): number { + return this.pending.size; + } + + private matchesRule(pending: PendingPermission): boolean { + const now = this.now(); + this.rules = this.rules.filter((rule) => rule.expiresAt > now); + const key = titleKeyOf(pending.title); + return this.rules.some( + (rule) => + rule.sessionHandle === pending.sessionHandle && rule.titleKey === key, + ); + } + + private remember(pending: PendingPermission): void { + this.rules.push({ + sessionHandle: pending.sessionHandle, + titleKey: titleKeyOf(pending.title), + expiresAt: this.now() + this.ruleTtlMs, + }); + while (this.rules.length > MAX_RULES) this.rules.shift(); + } + + private async deliver( + pending: PendingPermission, + decision: 'allow' | 'deny', + auto: boolean, + note?: string, + ): Promise<'delivered' | 'already_resolved'> { + const outcome = await this.options.adaptor.respondPermission( + pending.backend, + pending.requestId, + decision, + ); + this.options.log?.('permission.decision', { + requestHandle: pending.requestHandle, + requestId: pending.requestId, + decision, + auto, + outcome, + ...(note ? { note } : {}), + }); + if (outcome === 'delivered' || outcome === 'already_resolved') { + this.pending.delete(pending.requestHandle); + this.pendingByRequestId.delete(pending.requestId); + } + return outcome; + } +} diff --git a/packages/qwen-live/src/realtime/instructions.ts b/packages/qwen-live/src/realtime/instructions.ts new file mode 100644 index 00000000000..70b98c53b12 --- /dev/null +++ b/packages/qwen-live/src/realtime/instructions.ts @@ -0,0 +1,58 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Realtime model instructions for the orchestration tool surface. Rewritten + * from the PR #7859 single-backend prompt for the session-dispatch model: + * the voice model is the user's control tower over many coding sessions, + * not the front half of one bound session. + */ + +const DEFAULT_INSTRUCTIONS = `## Identity, tone, and role + +You are Qwen Code, a general-purpose agentic assistant. You are the user's single voice entry point to everything their coding sessions can do: files, commands, apps, documents, research, and long-running work. + +Be concise, clear, and efficient. Keep responses tight and useful—no fluff. Talk like a trusted collaborator: natural, warm, and easy to follow, with light energy and zero ceremony. + +## Operating model + +You coordinate coding sessions that do the actual work. The user cannot see your tools; present everything as done by you. Never mention "sessions", "backends", "tools", or how the system is put together unless the user asks about the machinery explicitly. + +* Anything that touches files, runs commands, needs current information, creates artifacts, or takes real action goes through \`handoff\`. When unsure whether a handoff would help, hand off. +* Respond directly only when the request is clearly self-contained conversation. +* NEVER refuse a request yourself, and never claim you lack an ability without trying. The executing session judges feasibility and safety; pass the request through with \`handoff\` and let it decide. +* When the user asks about the screen or visible content, call \`appshot\`; for anything deeper than describing what is visible, follow with a \`handoff\` and attach the capture. +* Multiple sessions may be working at once. \`session_list\` shows what exists; refer to sessions the way the user does ("the test one"), and use handles only as tool arguments, never aloud. + +## Receipts, results, and honesty + +* Tools return receipts and snapshots, never final results. A receipt means the work is queued or running — nothing more. +* Never say work is done, created, or successful without evidence: a receipt for "started", a [COMPLETE] message for "finished". If you have not seen it, say it is still in progress. +* Results arrive as [COMPLETE] or [PROGRESS] context messages. [BACKEND]-style context messages are silent context: never respond merely because one arrived. +* A [SPEAK_TO_USER] message is an explicit one-shot speech request: speak exactly the text after the prefix, verbatim, without additions or tool calls. +* Before your first tool call in a user turn, say one short, neutral sentence about what you are about to do ("Let me get that going."). Never promise outcomes in it. Then call the tool immediately. Do not repeat the acknowledgement for follow-up calls in the same turn. + +## Steering, stopping, and interruptions + +* New instructions, corrections, or constraints for running work: \`handoff\` to the same session immediately. Running work is always steerable — never claim otherwise. +* The user interrupting your speech never stops any work. Work stops only through \`session_stop\`, and only when the user clearly asks for that. + +## Permissions + +* A [PERMISSION] message means a session is waiting for the user's approval. Read it out briefly and plainly — what wants to run, in everyday words — and relay their answer with \`respond_permission\`. +* Never answer a permission request on the user's behalf, and never pressure them either way. + +## Presenting results + +* When a [COMPLETE] arrives at a natural moment, give the user the key takeaway in one or two spoken sentences: what happened, what changed, what needs them next. +* Do not read out tables, diffs, code, paths, or structured data. Offer the gist; the details are on their screen when they want them. +* Follow the user's stated preferences about update frequency and verbosity for the rest of the task.`; + +export function buildLiveInstructions(startupContext?: string): string { + return startupContext + ? `${DEFAULT_INSTRUCTIONS}\n\n${startupContext}` + : DEFAULT_INSTRUCTIONS; +} diff --git a/packages/qwen-live/src/realtime/realtime-session.test.ts b/packages/qwen-live/src/realtime/realtime-session.test.ts new file mode 100644 index 00000000000..3df3229bbcc --- /dev/null +++ b/packages/qwen-live/src/realtime/realtime-session.test.ts @@ -0,0 +1,1363 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it, vi } from 'vitest'; +import { + deriveQwenOmniRealtimeUrl, + openQwenRealtimeSession, + QWEN_REALTIME_LIMITS, + REMAIN_SILENT_TOOL_NAME, + type QwenRealtimeCallbacks, + type QwenRealtimeSession, + type RealtimeToolDefinition, +} from './realtime-session.js'; + +const HANDOFF_TOOL: RealtimeToolDefinition = { + type: 'function', + function: { + name: 'handoff', + description: 'Delegate the current request to the backend agent.', + parameters: { + type: 'object', + properties: { task: { type: 'string' } }, + required: ['task'], + additionalProperties: false, + }, + }, + capturesTranscript: true, +}; + +const LIST_TOOL: RealtimeToolDefinition = { + type: 'function', + function: { + name: 'session_list', + description: 'List the active backend sessions.', + parameters: { type: 'object', properties: {} }, + }, +}; + +const REMAIN_SILENT_DEF: RealtimeToolDefinition = { + type: 'function', + function: { + name: REMAIN_SILENT_TOOL_NAME, + description: 'Stay silent for this turn.', + parameters: { type: 'object', properties: {} }, + }, +}; + +class FakeSocket { + readonly OPEN = 1; + readyState = this.OPEN; + bufferedAmount = 0; + readonly sent: Array = []; + private readonly handlers = new Map< + string, + Array<(...args: unknown[]) => void> + >(); + + send(data: string | Uint8Array): void { + this.sent.push(data); + } + + close(): void { + this.readyState = 3; + } + + on(event: string, cb: (...args: unknown[]) => void): void { + const handlers = this.handlers.get(event) ?? []; + handlers.push(cb); + this.handlers.set(event, handlers); + } + + emit(event: string, ...args: unknown[]): void { + for (const handler of this.handlers.get(event) ?? []) handler(...args); + } + + message(body: Record): void { + this.emit('message', JSON.stringify(body), false); + } +} + +function sentJson(socket: FakeSocket, index: number): Record { + return JSON.parse(String(socket.sent[index])); +} + +function sentTypes(socket: FakeSocket): string[] { + return socket.sent.map((entry) => String(sentJsonEntry(entry)['type'])); +} + +function sentJsonEntry(entry: string | Uint8Array): Record { + return JSON.parse(String(entry)); +} + +function commitFinalInput( + socket: FakeSocket, + itemId: string, + transcript: string, +): void { + socket.message({ + type: 'input_audio_buffer.committed', + event_id: itemId + '-committed', + item_id: itemId, + }); + socket.message({ + type: 'conversation.item.input_audio_transcription.completed', + event_id: itemId + '-transcript', + item_id: itemId, + transcript, + }); +} + +function responseCreated(socket: FakeSocket, responseId: string): void { + socket.message({ + type: 'response.created', + event_id: responseId + '-created', + response: { id: responseId, status: 'in_progress' }, + }); +} + +function responseDone( + socket: FakeSocket, + responseId: string, + status = 'completed', +): void { + socket.message({ + type: 'response.done', + event_id: responseId + '-done', + response: { id: responseId, status }, + }); +} + +function functionCall( + socket: FakeSocket, + responseId: string, + callId: string, + name: string, + argumentsText: string, +): void { + socket.message({ + type: 'response.output_item.done', + event_id: callId + '-done', + response_id: responseId, + item: { + id: 'item-' + callId, + type: 'function_call', + name, + call_id: callId, + arguments: argumentsText, + }, + }); +} + +async function connect( + socket: FakeSocket, + callbacks: QwenRealtimeCallbacks = {}, +): Promise { + const opening = openQwenRealtimeSession( + { + endpoint: 'https://dashscope.example/compatible-mode/v1', + apiKey: 'sk-test', + model: 'qwen3.5-omni-plus-realtime', + callEpoch: 7, + voice: 'Tina', + instructions: 'test instructions', + tools: [HANDOFF_TOOL, LIST_TOOL, REMAIN_SILENT_DEF], + }, + callbacks, + { createWebSocket: () => socket }, + ); + socket.message({ type: 'session.created', event_id: 'session-created' }); + socket.message({ + type: 'session.updated', + event_id: 'session-updated', + session: { id: 'session-1' }, + }); + return opening; +} + +describe('realtime-session', () => { + it('derives a model-qualified WebSocket URL', () => { + expect( + deriveQwenOmniRealtimeUrl( + 'https://dashscope.aliyuncs.com/compatible-mode/v1', + 'qwen3.5-omni-plus-realtime', + ), + ).toBe( + 'wss://dashscope.aliyuncs.com/api-ws/v1/realtime?model=qwen3.5-omni-plus-realtime', + ); + expect( + deriveQwenOmniRealtimeUrl( + 'wss://example.test/custom/api-ws/v1/realtime?tenant=one', + 'model/with spaces', + ), + ).toBe( + 'wss://example.test/custom/api-ws/v1/realtime?tenant=one&model=model%2Fwith+spaces', + ); + }); + + it('sends the configured instructions and wire-shaped tools in session.update', async () => { + const socket = new FakeSocket(); + await connect(socket); + + const update = sentJson(socket, 0); + expect(update['type']).toBe('session.update'); + const session = update['session'] as Record; + expect(session['modalities']).toEqual(['text', 'audio']); + expect(session['voice']).toBe('Tina'); + expect(session['tool_choice']).toBe('auto'); + expect(session['instructions']).toBe('test instructions'); + expect(session['turn_detection']).toEqual({ + type: 'semantic_vad', + create_response: true, + interrupt_response: true, + }); + // The wire shape carries only {type, function}: the local-only + // `capturesTranscript` flag must be stripped. + expect(session['tools']).toEqual([ + { type: 'function', function: HANDOFF_TOOL.function }, + { type: 'function', function: LIST_TOOL.function }, + { type: 'function', function: REMAIN_SILENT_DEF.function }, + ]); + for (const tool of session['tools'] as Array>) { + expect('capturesTranscript' in tool).toBe(false); + } + }); + + it('lets Realtime answer an ordinary turn directly', async () => { + const socket = new FakeSocket(); + const callbacks = { + onFunctionCall: vi.fn(), + onResponseCreated: vi.fn(), + onOutputTextDelta: vi.fn(), + onOutputTextDone: vi.fn(), + onOutputAudioDelta: vi.fn(), + onOutputAudioDone: vi.fn(), + onResponseDone: vi.fn(), + onDirectTranscript: vi.fn(), + } satisfies QwenRealtimeCallbacks; + const session = await connect(socket, callbacks); + + commitFinalInput(socket, 'input-direct', '你好'); + responseCreated(socket, 'response-direct'); + socket.message({ + type: 'response.audio_transcript.delta', + response_id: 'response-direct', + item_id: 'assistant-direct', + delta: '你好！', + }); + socket.message({ + type: 'response.audio_transcript.done', + response_id: 'response-direct', + item_id: 'assistant-direct', + transcript: '你好！', + }); + socket.message({ + type: 'response.output_audio.delta', + response_id: 'response-direct', + item_id: 'assistant-direct', + delta: Buffer.from([1, 0, 2, 0]).toString('base64'), + }); + socket.message({ + type: 'response.output_audio.done', + response_id: 'response-direct', + item_id: 'assistant-direct', + }); + responseDone(socket, 'response-direct'); + + expect(callbacks.onFunctionCall).not.toHaveBeenCalled(); + expect(callbacks.onResponseCreated).toHaveBeenCalledWith( + expect.objectContaining({ + responseId: 'response-direct', + inputItemId: 'input-direct', + authority: 'direct', + }), + ); + expect(callbacks.onOutputTextDone).toHaveBeenCalledWith( + expect.objectContaining({ text: '你好！', source: 'audio_transcript' }), + ); + expect(callbacks.onOutputAudioDelta).toHaveBeenCalledWith( + expect.objectContaining({ audio: new Uint8Array([1, 0, 2, 0]) }), + ); + expect(callbacks.onResponseDone).toHaveBeenCalledWith( + expect.objectContaining({ responseId: 'response-direct' }), + ); + expect(callbacks.onDirectTranscript).toHaveBeenCalledWith({ + callEpoch: 7, + entries: [ + { role: 'user', text: '你好' }, + { role: 'assistant', text: '你好！' }, + ], + }); + expect(session.takeTranscriptTail()).toEqual([]); + expect(sentTypes(socket)).toEqual(['session.update']); + }); + + it('returns undelivered direct dialogue once when no transcript callback is configured', async () => { + const socket = new FakeSocket(); + const session = await connect(socket); + + commitFinalInput(socket, 'input-direct', '你好'); + responseCreated(socket, 'response-direct'); + socket.message({ + type: 'response.audio_transcript.done', + response_id: 'response-direct', + transcript: '你好！', + }); + responseDone(socket, 'response-direct'); + + expect(session.takeTranscriptTail()).toEqual([ + { role: 'user', text: '你好' }, + { role: 'assistant', text: '你好！' }, + ]); + expect(session.takeTranscriptTail()).toEqual([]); + }); + + it('uses the active response when response.done omits its identifier', async () => { + const socket = new FakeSocket(); + const callbacks = { + onError: vi.fn(), + onResponseDone: vi.fn(), + } satisfies QwenRealtimeCallbacks; + await connect(socket, callbacks); + + commitFinalInput(socket, 'input-direct', '你好'); + responseCreated(socket, 'response-direct'); + socket.message({ + type: 'response.done', + event_id: 'response-direct-done', + response: { status: 'completed' }, + }); + + expect(callbacks.onError).not.toHaveBeenCalled(); + expect(callbacks.onResponseDone).toHaveBeenCalledWith( + expect.objectContaining({ + responseId: 'response-direct', + inputItemId: 'input-direct', + status: 'completed', + }), + ); + }); + + it('ignores an idless duplicate response.done after a completed response', async () => { + const socket = new FakeSocket(); + const callbacks = { + onError: vi.fn(), + onIgnoredEvent: vi.fn(), + onResponseDone: vi.fn(), + onFunctionCall: vi.fn(), + } satisfies QwenRealtimeCallbacks; + await connect(socket, callbacks); + + commitFinalInput(socket, 'input-direct', '你好'); + responseCreated(socket, 'response-direct'); + responseDone(socket, 'response-direct'); + socket.message({ + type: 'response.done', + event_id: 'response-direct-late-done', + response: { status: 'completed' }, + }); + + expect(callbacks.onError).not.toHaveBeenCalled(); + expect(callbacks.onResponseDone).toHaveBeenCalledOnce(); + expect(callbacks.onIgnoredEvent).toHaveBeenCalledWith( + expect.objectContaining({ + type: 'response.done', + reason: 'stale_response', + }), + ); + + commitFinalInput(socket, 'input-handoff', '检查当前页面'); + responseCreated(socket, 'response-handoff'); + functionCall( + socket, + 'response-handoff', + 'call-handoff', + 'handoff', + JSON.stringify({ task: '检查当前页面' }), + ); + expect(callbacks.onFunctionCall).toHaveBeenCalledOnce(); + }); + + it('rejects an idless response.done when no response has completed', async () => { + const socket = new FakeSocket(); + const callbacks = { onError: vi.fn() } satisfies QwenRealtimeCallbacks; + const session = await connect(socket, callbacks); + + socket.message({ + type: 'response.done', + event_id: 'orphan-response-done', + response: { status: 'completed' }, + }); + + expect(callbacks.onError).toHaveBeenCalledWith( + expect.objectContaining({ code: 'invalid_response' }), + ); + await expect(session.closed).resolves.toMatchObject({ reason: 'error' }); + }); + + it('dispatches a capturing tool call and holds its output until response.done', async () => { + const socket = new FakeSocket(); + const callbacks = { onFunctionCall: vi.fn() }; + const session = await connect(socket, callbacks); + + commitFinalInput(socket, 'input-handoff', '查看当前仓库'); + responseCreated(socket, 'response-handoff'); + functionCall( + socket, + 'response-handoff', + 'call-handoff', + 'handoff', + JSON.stringify({ task: '查看当前仓库' }), + ); + + expect(callbacks.onFunctionCall).toHaveBeenCalledWith( + expect.objectContaining({ + callEpoch: 7, + responseId: 'response-handoff', + inputItemId: 'input-handoff', + callId: 'call-handoff', + name: 'handoff', + arguments: JSON.stringify({ task: '查看当前仓库' }), + activeTranscript: [{ role: 'user', text: '查看当前仓库' }], + }), + ); + const event = callbacks.onFunctionCall.mock.calls[0]![0] as { + arguments: string; + }; + expect(JSON.parse(event.arguments)).toEqual({ task: '查看当前仓库' }); + // The captured turn is delegated and stays out of the direct collection. + expect(session.takeTranscriptTail()).toEqual([]); + + // Output submitted before the response completes is held back... + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-handoff' }, + '仓库检查完成。', + ), + ).toBe(true); + expect(sentTypes(socket)).toEqual(['session.update']); + + // ...and flushed right after response.done. + responseDone(socket, 'response-handoff'); + await Promise.resolve(); + expect(sentTypes(socket)).toEqual([ + 'session.update', + 'conversation.item.create', + ]); + expect(sentJson(socket, 1)['item']).toEqual({ + type: 'function_call_output', + call_id: 'call-handoff', + output: '仓库检查完成。', + }); + }); + + it('dispatches multiple function calls in one response independently', async () => { + const socket = new FakeSocket(); + const callbacks = { onFunctionCall: vi.fn() }; + const session = await connect(socket, callbacks); + + commitFinalInput(socket, 'input-multi', '并行处理'); + responseCreated(socket, 'response-multi'); + functionCall( + socket, + 'response-multi', + 'call-a', + 'handoff', + JSON.stringify({ task: '并行处理' }), + ); + functionCall(socket, 'response-multi', 'call-b', 'session_list', '{}'); + + expect(callbacks.onFunctionCall).toHaveBeenCalledTimes(2); + expect(callbacks.onFunctionCall).toHaveBeenNthCalledWith( + 1, + expect.objectContaining({ + callId: 'call-a', + name: 'handoff', + activeTranscript: [{ role: 'user', text: '并行处理' }], + }), + ); + expect(callbacks.onFunctionCall).toHaveBeenNthCalledWith( + 2, + expect.objectContaining({ + callId: 'call-b', + name: 'session_list', + arguments: '{}', + activeTranscript: [], + }), + ); + + responseDone(socket, 'response-multi'); + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-a' }, + 'A 完成', + ), + ).toBe(true); + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-b' }, + 'B 完成', + ), + ).toBe(true); + + const outputs = socket.sent + .map(sentJsonEntry) + .map((entry) => entry['item'] as Record | undefined) + .filter((item) => item?.['type'] === 'function_call_output'); + expect(outputs).toEqual([ + { type: 'function_call_output', call_id: 'call-a', output: 'A 完成' }, + { type: 'function_call_output', call_id: 'call-b', output: 'B 完成' }, + ]); + }); + + it('keeps non-capturing tool turns inside the direct transcript flow', async () => { + const socket = new FakeSocket(); + const callbacks = { onFunctionCall: vi.fn() }; + const session = await connect(socket, callbacks); + + commitFinalInput(socket, 'input-list', '列出会话'); + responseCreated(socket, 'response-list'); + functionCall(socket, 'response-list', 'call-list', 'session_list', '{}'); + + expect(callbacks.onFunctionCall).toHaveBeenCalledWith( + expect.objectContaining({ + callId: 'call-list', + name: 'session_list', + arguments: '{}', + activeTranscript: [], + }), + ); + + socket.message({ + type: 'response.audio_transcript.done', + response_id: 'response-list', + transcript: '好的，这是会话列表。', + }); + responseDone(socket, 'response-list'); + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-list' }, + '会话：alpha', + ), + ).toBe(true); + + // The response stayed `direct`, so its dialogue is still collected. + expect(session.takeTranscriptTail()).toEqual([ + { role: 'user', text: '列出会话' }, + { role: 'assistant', text: '好的，这是会话列表。' }, + ]); + }); + + it('handles remain_silent without dispatching a function call', async () => { + const socket = new FakeSocket(); + const callbacks = { onFunctionCall: vi.fn() }; + await connect(socket, callbacks); + + commitFinalInput(socket, 'input-silent', ''); + responseCreated(socket, 'response-silent'); + functionCall( + socket, + 'response-silent', + 'call-silent', + REMAIN_SILENT_TOOL_NAME, + '{}', + ); + responseDone(socket, 'response-silent'); + await Promise.resolve(); + + expect(callbacks.onFunctionCall).not.toHaveBeenCalled(); + expect(sentJson(socket, 1)['item']).toEqual({ + type: 'function_call_output', + call_id: 'call-silent', + output: '', + }); + expect(sentTypes(socket)).toEqual([ + 'session.update', + 'conversation.item.create', + ]); + }); + + it('answers unknown tools with an error receipt instead of dispatching them', async () => { + const socket = new FakeSocket(); + const callbacks = { + onFunctionCall: vi.fn(), + onError: vi.fn(), + } satisfies QwenRealtimeCallbacks; + const session = await connect(socket, callbacks); + + commitFinalInput(socket, 'input-screen', '查看屏幕'); + responseCreated(socket, 'response-screen'); + functionCall( + socket, + 'response-screen', + 'call-screen', + 'handoff', + JSON.stringify({ task: '查看屏幕' }), + ); + functionCall(socket, 'response-screen', 'call-unknown', 'appshot', '{}'); + responseDone(socket, 'response-screen'); + await Promise.resolve(); + + expect(callbacks.onFunctionCall).toHaveBeenCalledTimes(1); + expect(callbacks.onError).not.toHaveBeenCalled(); + expect(sentJson(socket, 1)['item']).toEqual({ + type: 'function_call_output', + call_id: 'call-unknown', + output: JSON.stringify({ + status: 'error', + note: 'Unknown tool: appshot', + }), + }); + // The declared call in the same response is still completable. + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-screen' }, + '屏幕已读取。', + ), + ).toBe(true); + }); + + it('rejects stale function outputs without sending anything', async () => { + const socket = new FakeSocket(); + const callbacks = { + onIgnoredEvent: vi.fn(), + onFunctionCall: vi.fn(), + } satisfies QwenRealtimeCallbacks; + const session = await connect(socket, callbacks); + + // Unknown call id. + expect( + session.submitFunctionOutput({ callEpoch: 7, callId: 'missing' }, '内容'), + ).toBe(false); + expect(callbacks.onIgnoredEvent).toHaveBeenCalledWith( + expect.objectContaining({ + type: 'conversation.item.create', + reason: 'stale_call', + }), + ); + + commitFinalInput(socket, 'input-stale', '检查'); + responseCreated(socket, 'response-stale'); + functionCall( + socket, + 'response-stale', + 'call-stale', + 'handoff', + JSON.stringify({ task: '检查' }), + ); + // Wrong epoch. + expect( + session.submitFunctionOutput( + { callEpoch: 8, callId: 'call-stale' }, + '内容', + ), + ).toBe(false); + // A call whose arguments are still streaming has not been dispatched. + socket.message({ + type: 'response.function_call_arguments.delta', + event_id: 'partial-delta', + response_id: 'response-stale', + call_id: 'call-partial', + delta: '{"task"', + }); + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-partial' }, + '内容', + ), + ).toBe(false); + + responseDone(socket, 'response-stale'); + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-stale' }, + '完成', + ), + ).toBe(true); + // Double submission. + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-stale' }, + '再次', + ), + ).toBe(false); + expect(callbacks.onIgnoredEvent).toHaveBeenCalledTimes(4); + + const outputs = socket.sent + .map(sentJsonEntry) + .map((entry) => entry['item'] as Record | undefined) + .filter((item) => item?.['type'] === 'function_call_output'); + expect(outputs).toEqual([ + { type: 'function_call_output', call_id: 'call-stale', output: '完成' }, + ]); + }); + + it('rejects function outputs after the session is closed', async () => { + const socket = new FakeSocket(); + const callbacks = { + onIgnoredEvent: vi.fn(), + onFunctionCall: vi.fn(), + } satisfies QwenRealtimeCallbacks; + const session = await connect(socket, callbacks); + + commitFinalInput(socket, 'input-late', '检查'); + responseCreated(socket, 'response-late'); + functionCall( + socket, + 'response-late', + 'call-late', + 'handoff', + JSON.stringify({ task: '检查' }), + ); + session.close(); + + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-late' }, + '完成', + ), + ).toBe(false); + expect(callbacks.onIgnoredEvent).toHaveBeenCalledWith( + expect.objectContaining({ reason: 'stale_call' }), + ); + }); + + it('bounds function output size and rejects blank output', async () => { + const socket = new FakeSocket(); + const session = await connect(socket, { onFunctionCall: vi.fn() }); + + commitFinalInput(socket, 'input-bounds', '检查'); + responseCreated(socket, 'response-bounds'); + functionCall( + socket, + 'response-bounds', + 'call-bounds', + 'handoff', + JSON.stringify({ task: '检查' }), + ); + responseDone(socket, 'response-bounds'); + + const ref = { callEpoch: 7, callId: 'call-bounds' }; + expect(() => session.submitFunctionOutput(ref, '')).toThrow(RangeError); + expect(() => session.submitFunctionOutput(ref, ' ')).toThrow(RangeError); + expect(() => + session.submitFunctionOutput( + ref, + 'x'.repeat(QWEN_REALTIME_LIMITS.maxFunctionOutputChars + 1), + ), + ).toThrow(RangeError); + // The call survives rejected attempts and can still be completed. + expect(session.submitFunctionOutput(ref, '完成')).toBe(true); + }); + + it('continues direct Realtime conversation after a handoff completes', async () => { + const socket = new FakeSocket(); + const callbacks = { + onFunctionCall: vi.fn(), + onResponseCreated: vi.fn(), + onOutputTextDone: vi.fn(), + onDirectTranscript: vi.fn(), + } satisfies QwenRealtimeCallbacks; + const session = await connect(socket, callbacks); + + commitFinalInput(socket, 'input-work', '执行任务'); + responseCreated(socket, 'response-work'); + functionCall( + socket, + 'response-work', + 'call-work', + 'handoff', + JSON.stringify({ task: '执行任务' }), + ); + responseDone(socket, 'response-work'); + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-work' }, + '任务完成。', + ), + ).toBe(true); + session.speakToUser('任务完成。'); + responseCreated(socket, 'response-work-result'); + socket.message({ + type: 'response.audio_transcript.done', + response_id: 'response-work-result', + transcript: '任务完成。', + }); + responseDone(socket, 'response-work-result'); + + commitFinalInput(socket, 'input-chat', '谢谢'); + responseCreated(socket, 'response-chat'); + socket.message({ + type: 'response.audio_transcript.done', + response_id: 'response-chat', + transcript: '不客气。', + }); + responseDone(socket, 'response-chat'); + + expect(callbacks.onFunctionCall).toHaveBeenCalledTimes(1); + expect(callbacks.onResponseCreated).toHaveBeenLastCalledWith( + expect.objectContaining({ + responseId: 'response-chat', + inputItemId: 'input-chat', + authority: 'direct', + }), + ); + expect(callbacks.onOutputTextDone).toHaveBeenLastCalledWith( + expect.objectContaining({ text: '不客气。' }), + ); + expect(callbacks.onDirectTranscript).toHaveBeenCalledOnce(); + expect(callbacks.onDirectTranscript).toHaveBeenCalledWith({ + callEpoch: 7, + entries: [ + { role: 'user', text: '谢谢' }, + { role: 'assistant', text: '不客气。' }, + ], + }); + }); + + it('keeps the replacement response alive when VAD interrupts audio', async () => { + const socket = new FakeSocket(); + const callbacks = { + onBargeIn: vi.fn(), + onError: vi.fn(), + onResponseCreated: vi.fn(), + onResponseDone: vi.fn(), + onOutputAudioDelta: vi.fn(), + onOutputTextDone: vi.fn(), + } satisfies QwenRealtimeCallbacks; + await connect(socket, callbacks); + + commitFinalInput(socket, 'input-first', '先回答第一个问题'); + responseCreated(socket, 'response-first'); + socket.message({ + type: 'response.output_audio.delta', + response_id: 'response-first', + item_id: 'assistant-first', + delta: Buffer.from([1, 0]).toString('base64'), + }); + + socket.message({ + type: 'input_audio_buffer.speech_started', + event_id: 'speech-started', + item_id: 'input-second', + }); + commitFinalInput(socket, 'input-second', '现在回答第二个问题'); + responseCreated(socket, 'response-second'); + responseDone(socket, 'response-first', 'cancelled'); + socket.message({ + type: 'response.output_audio.delta', + response_id: 'response-second', + item_id: 'assistant-second', + delta: Buffer.from([2, 0]).toString('base64'), + }); + socket.message({ + type: 'response.audio_transcript.done', + response_id: 'response-second', + item_id: 'assistant-second', + transcript: '第二个问题的回答。', + }); + responseDone(socket, 'response-second'); + + expect(callbacks.onBargeIn).toHaveBeenCalledWith( + expect.objectContaining({ responseId: 'response-first' }), + ); + expect(sentTypes(socket)).not.toContain('response.cancel'); + expect(callbacks.onError).not.toHaveBeenCalled(); + expect(callbacks.onResponseCreated).toHaveBeenLastCalledWith( + expect.objectContaining({ + responseId: 'response-second', + inputItemId: 'input-second', + authority: 'direct', + }), + ); + expect(callbacks.onResponseDone).toHaveBeenNthCalledWith( + 1, + expect.objectContaining({ + responseId: 'response-first', + status: 'cancelled', + }), + ); + expect(callbacks.onOutputAudioDelta).toHaveBeenLastCalledWith( + expect.objectContaining({ + responseId: 'response-second', + audio: new Uint8Array([2, 0]), + }), + ); + expect(callbacks.onOutputTextDone).toHaveBeenLastCalledWith( + expect.objectContaining({ + responseId: 'response-second', + text: '第二个问题的回答。', + }), + ); + expect(callbacks.onResponseDone).toHaveBeenLastCalledWith( + expect.objectContaining({ + responseId: 'response-second', + status: 'completed', + }), + ); + }); + + it('finalizes a superseded response when the provider omits its done event', async () => { + const socket = new FakeSocket(); + const callbacks = { + onError: vi.fn(), + onResponseDone: vi.fn(), + onDirectTranscript: vi.fn(), + } satisfies QwenRealtimeCallbacks; + const session = await connect(socket, callbacks); + + commitFinalInput(socket, 'input-first', '第一个问题'); + responseCreated(socket, 'response-first'); + socket.message({ + type: 'response.audio_transcript.done', + response_id: 'response-first', + transcript: '第一个回答。', + }); + + commitFinalInput(socket, 'input-second', '第二个问题'); + responseCreated(socket, 'response-second'); + socket.message({ + type: 'response.audio_transcript.done', + response_id: 'response-second', + transcript: '第二个回答。', + }); + responseDone(socket, 'response-second'); + + expect(callbacks.onError).not.toHaveBeenCalled(); + expect(callbacks.onResponseDone).toHaveBeenNthCalledWith( + 1, + expect.objectContaining({ + responseId: 'response-first', + inputItemId: 'input-first', + status: 'cancelled', + }), + ); + expect(callbacks.onResponseDone).toHaveBeenNthCalledWith( + 2, + expect.objectContaining({ + responseId: 'response-second', + inputItemId: 'input-second', + status: 'completed', + }), + ); + expect(callbacks.onDirectTranscript).toHaveBeenNthCalledWith(1, { + callEpoch: 7, + entries: [ + { role: 'user', text: '第一个问题' }, + { role: 'assistant', text: '第一个回答。' }, + ], + }); + expect(session.takeTranscriptTail()).toEqual([]); + }); + + it('keeps delegated work alive when its response is superseded without done', async () => { + const socket = new FakeSocket(); + const callbacks = { + onFunctionCall: vi.fn(), + onResponseDone: vi.fn(), + onDirectTranscript: vi.fn(), + } satisfies QwenRealtimeCallbacks; + const session = await connect(socket, callbacks); + + commitFinalInput(socket, 'input-handoff', '检查当前页面'); + responseCreated(socket, 'response-handoff'); + functionCall( + socket, + 'response-handoff', + 'call-handoff', + 'handoff', + JSON.stringify({ task: '检查当前页面' }), + ); + + commitFinalInput(socket, 'input-next', '谢谢'); + responseCreated(socket, 'response-next'); + + expect(callbacks.onResponseDone).toHaveBeenCalledWith( + expect.objectContaining({ + responseId: 'response-handoff', + inputItemId: 'input-handoff', + status: 'cancelled', + }), + ); + expect(callbacks.onDirectTranscript).not.toHaveBeenCalled(); + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-handoff' }, + '页面检查完成。', + ), + ).toBe(true); + expect(sentTypes(socket)).not.toContain('response.create'); + + responseDone(socket, 'response-next'); + expect(session.takeTranscriptTail()).toEqual([]); + }); + + it('keeps backend context silent while a response is active', async () => { + const socket = new FakeSocket(); + const session = await connect(socket); + + commitFinalInput(socket, 'input-active', '你好'); + responseCreated(socket, 'response-active'); + expect(session.sendBackendContext('后台消息一')).toBe(true); + expect(session.sendBackendContext('后台消息二')).toBe(true); + expect( + sentTypes(socket).filter((type) => type === 'response.create'), + ).toHaveLength(0); + expect(sentJson(socket, 1)['item']).toEqual({ + type: 'message', + role: 'user', + content: [{ type: 'input_text', text: '[BACKEND] 后台消息一' }], + }); + expect(sentJson(socket, 2)['item']).toEqual({ + type: 'message', + role: 'user', + content: [{ type: 'input_text', text: '[BACKEND] 后台消息二' }], + }); + + responseDone(socket, 'response-active'); + await Promise.resolve(); + expect( + sentTypes(socket).filter((type) => type === 'response.create'), + ).toHaveLength(0); + }); + + it('speaks only explicit backend speech with backend_speech authority', async () => { + const socket = new FakeSocket(); + const callbacks = { onResponseCreated: vi.fn() }; + const session = await connect(socket, callbacks); + + expect(session.sendBackendContext('静默上下文')).toBe(true); + expect(session.speakToUser('正在检查，请稍等。')).toBe(true); + expect(sentTypes(socket)).toEqual([ + 'session.update', + 'conversation.item.create', + 'conversation.item.create', + 'response.create', + ]); + expect(sentJson(socket, 2)['item']).toEqual({ + type: 'message', + role: 'user', + content: [ + { + type: 'input_text', + text: '[SPEAK_TO_USER] 正在检查，请稍等。', + }, + ], + }); + expect(sentJson(socket, 3)).toMatchObject({ + type: 'response.create', + response: { modalities: ['text', 'audio'] }, + }); + + responseCreated(socket, 'response-speech'); + expect(callbacks.onResponseCreated).toHaveBeenCalledWith( + expect.objectContaining({ + responseId: 'response-speech', + authority: 'backend_speech', + }), + ); + }); + + it('drops queued backend speech when the user starts speaking', async () => { + const socket = new FakeSocket(); + const session = await connect(socket); + + commitFinalInput(socket, 'input-active', '你好'); + responseCreated(socket, 'response-active'); + expect(session.speakToUser('旧进度一')).toBe(true); + expect(session.speakToUser('旧进度二')).toBe(true); + socket.message({ + type: 'input_audio_buffer.speech_started', + event_id: 'speech-started-new', + item_id: 'input-new', + }); + responseDone(socket, 'response-active'); + await Promise.resolve(); + + expect( + sentTypes(socket).filter((type) => type === 'response.create'), + ).toHaveLength(0); + }); + + it('serializes multiple explicit speech requests without combining them', async () => { + const socket = new FakeSocket(); + const session = await connect(socket); + + commitFinalInput(socket, 'input-active', '你好'); + responseCreated(socket, 'response-active'); + expect(session.speakToUser('第一条')).toBe(true); + expect(session.speakToUser('第二条')).toBe(true); + expect( + sentTypes(socket).filter((type) => type === 'conversation.item.create'), + ).toHaveLength(0); + + responseDone(socket, 'response-active'); + await Promise.resolve(); + responseCreated(socket, 'response-first-speech'); + responseDone(socket, 'response-first-speech'); + await Promise.resolve(); + + const speechItems = socket.sent + .map(sentJsonEntry) + .filter((entry) => entry['type'] === 'conversation.item.create') + .map((entry) => entry['item']); + expect(speechItems).toEqual([ + { + type: 'message', + role: 'user', + content: [{ type: 'input_text', text: '[SPEAK_TO_USER] 第一条' }], + }, + { + type: 'message', + role: 'user', + content: [{ type: 'input_text', text: '[SPEAK_TO_USER] 第二条' }], + }, + ]); + expect( + sentTypes(socket).filter((type) => type === 'response.create'), + ).toHaveLength(2); + }); + + it('cancels backend speech requested just before user speech', async () => { + const socket = new FakeSocket(); + const callbacks = { onResponseCreated: vi.fn() }; + const session = await connect(socket, callbacks); + + expect(session.speakToUser('即将过期的进度')).toBe(true); + socket.message({ + type: 'input_audio_buffer.speech_started', + event_id: 'speech-started-before-created', + item_id: 'input-new', + }); + responseCreated(socket, 'response-stale-speech'); + + expect(sentTypes(socket)).toContain('response.cancel'); + expect(callbacks.onResponseCreated).not.toHaveBeenCalled(); + }); + + it('preserves audio backpressure and frame bounds', async () => { + const socket = new FakeSocket(); + const onAudioDropped = vi.fn(); + const session = await connect(socket, { onAudioDropped }); + + expect(() => + session.pushAudio( + new Uint8Array(QWEN_REALTIME_LIMITS.maxInputAudioFrameBytes + 2), + ), + ).toThrow(RangeError); + socket.bufferedAmount = QWEN_REALTIME_LIMITS.maxBufferedSocketBytes + 1; + expect(session.pushAudio(new Uint8Array([1, 0]))).toBe(false); + expect(session.pushAudio(new Uint8Array([1, 0]))).toBe(false); + expect(onAudioDropped).toHaveBeenCalledTimes(1); + }); + + it('redacts provider credentials and classifies rate limits', async () => { + const socket = new FakeSocket(); + const onError = vi.fn(); + const session = await connect(socket, { onError }); + socket.message({ + type: 'error', + error: { + code: 'rate_limit_exceeded', + status: 429, + message: 'sk-test rate limit exceeded', + }, + }); + + const closed = await session.closed; + expect(closed.reason).toBe('error'); + expect(closed.error?.kind).toBe('transient'); + expect(closed.error?.message).not.toContain('sk-test'); + expect(onError).toHaveBeenCalledOnce(); + }); + + it('rejects realtime endpoints carrying credentials', () => { + expect(() => + deriveQwenOmniRealtimeUrl( + 'https://user:pass@dashscope.example/compatible-mode/v1', + 'qwen3.5-omni-plus-realtime', + ), + ).toThrow('must not contain credentials'); + expect(() => + deriveQwenOmniRealtimeUrl( + 'https://dashscope.example/compatible-mode/v1?api_key=sk-secret', + 'qwen3.5-omni-plus-realtime', + ), + ).toThrow('must not contain credentials'); + expect(() => + deriveQwenOmniRealtimeUrl( + 'wss://dashscope.example/api-ws/v1/realtime?token=abc', + 'qwen3.5-omni-plus-realtime', + ), + ).toThrow('must not contain credentials'); + }); + + it('ignores a late final transcript for an already-consumed input', async () => { + const socket = new FakeSocket(); + const callbacks = { + onError: vi.fn(), + onIgnoredEvent: vi.fn(), + onInputTranscriptDone: vi.fn(), + } satisfies QwenRealtimeCallbacks; + await connect(socket, callbacks); + + // The turn completes before the ASR stream delivers its final. + socket.message({ + type: 'input_audio_buffer.committed', + event_id: 'late-committed', + item_id: 'input-late', + }); + responseCreated(socket, 'response-late'); + responseDone(socket, 'response-late'); + socket.message({ + type: 'conversation.item.input_audio_transcription.completed', + event_id: 'late-final', + item_id: 'input-late', + transcript: '迟到的转写', + }); + + expect(callbacks.onError).not.toHaveBeenCalled(); + expect(callbacks.onInputTranscriptDone).not.toHaveBeenCalled(); + expect(callbacks.onIgnoredEvent).toHaveBeenCalledWith( + expect.objectContaining({ + type: 'conversation.item.input_audio_transcription.completed', + reason: 'stale_input', + }), + ); + // The session survives and the next turn proceeds normally. + expect(socket.readyState).toBe(socket.OPEN); + commitFinalInput(socket, 'input-next', '下一个问题'); + expect(callbacks.onInputTranscriptDone).toHaveBeenCalledWith( + expect.objectContaining({ itemId: 'input-next', text: '下一个问题' }), + ); + expect(callbacks.onError).not.toHaveBeenCalled(); + }); + + it('still fails a final transcript for a never-committed input', async () => { + const socket = new FakeSocket(); + const callbacks = { onError: vi.fn() } satisfies QwenRealtimeCallbacks; + const session = await connect(socket, callbacks); + + socket.message({ + type: 'conversation.item.input_audio_transcription.completed', + event_id: 'orphan-final', + item_id: 'input-unknown', + transcript: '幽灵转写', + }); + + expect(callbacks.onError).toHaveBeenCalledWith( + expect.objectContaining({ code: 'unattributed_final_transcript' }), + ); + await expect(session.closed).resolves.toMatchObject({ reason: 'error' }); + }); + + it('re-captures repeated user input once handed-off entries are trimmed', async () => { + const socket = new FakeSocket(); + const callbacks = { onFunctionCall: vi.fn() }; + const session = await connect(socket, callbacks); + + commitFinalInput(socket, 'input-repeat-1', '重复的任务'); + responseCreated(socket, 'response-repeat-1'); + functionCall( + socket, + 'response-repeat-1', + 'call-repeat-1', + 'handoff', + JSON.stringify({ task: '重复的任务' }), + ); + expect(callbacks.onFunctionCall).toHaveBeenNthCalledWith( + 1, + expect.objectContaining({ + activeTranscript: [{ role: 'user', text: '重复的任务' }], + }), + ); + responseDone(socket, 'response-repeat-1'); + expect( + session.submitFunctionOutput( + { callEpoch: 7, callId: 'call-repeat-1' }, + '完成', + ), + ).toBe(true); + + // Second turn: the model calls the tool before any ASR events land, and + // the task text repeats the previous (already handed-off) user entry. + // The consumed entries were trimmed, so the repeat is captured again + // instead of being deduplicated against dead history. + socket.message({ + type: 'input_audio_buffer.committed', + event_id: 'repeat-2-committed', + item_id: 'input-repeat-2', + }); + responseCreated(socket, 'response-repeat-2'); + functionCall( + socket, + 'response-repeat-2', + 'call-repeat-2', + 'handoff', + JSON.stringify({ task: '重复的任务' }), + ); + expect(callbacks.onFunctionCall).toHaveBeenNthCalledWith( + 2, + expect.objectContaining({ + activeTranscript: [{ role: 'user', text: '重复的任务' }], + }), + ); + }); + + it('caps the retained handoff transcript instead of growing unboundedly', async () => { + const socket = new FakeSocket(); + const callbacks = { onFunctionCall: vi.fn() }; + await connect(socket, callbacks); + + // 300 direct turns append 600 transcript entries with no handoff to + // consume them. + for (let i = 0; i < 300; i += 1) { + commitFinalInput(socket, `input-${i}`, `问题 ${i}`); + responseCreated(socket, `response-${i}`); + socket.message({ + type: 'response.audio_transcript.done', + event_id: `response-${i}-transcript`, + response_id: `response-${i}`, + transcript: `回答 ${i}`, + }); + responseDone(socket, `response-${i}`); + } + + commitFinalInput(socket, 'input-handoff', '最后的任务'); + responseCreated(socket, 'response-handoff'); + functionCall( + socket, + 'response-handoff', + 'call-handoff', + 'handoff', + JSON.stringify({ task: '最后的任务' }), + ); + + const event = callbacks.onFunctionCall.mock.calls[0]![0] as { + activeTranscript: ReadonlyArray<{ role: string; text: string }>; + }; + // 601 entries were appended; only the newest 512 are retained. + expect(event.activeTranscript.length).toBe(512); + expect(event.activeTranscript.at(-1)).toEqual({ + role: 'user', + text: '最后的任务', + }); + }); + + it('distinguishes client and remote closure', async () => { + const clientSocket = new FakeSocket(); + const client = await connect(clientSocket); + client.close(); + await expect(client.closed).resolves.toEqual({ reason: 'client' }); + + const remoteSocket = new FakeSocket(); + const remote = await connect(remoteSocket); + remoteSocket.emit('close', 1007, Buffer.from('invalid request')); + const closed = await remote.closed; + expect(closed.reason).toBe('remote'); + expect(closed.error?.closeCode).toBe(1007); + }); +}); diff --git a/packages/qwen-live/src/realtime/realtime-session.ts b/packages/qwen-live/src/realtime/realtime-session.ts new file mode 100644 index 00000000000..125cf10c662 --- /dev/null +++ b/packages/qwen-live/src/realtime/realtime-session.ts @@ -0,0 +1,2195 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * DashScope qwen-omni realtime client, ported from + * packages/cli/src/serve/live/qwen-realtime-session.ts. + * + * The transport, response-arbitration, and defensive-validation layers are + * unchanged from the port. What changed for the standalone live daemon is the + * tool surface: instead of a single hardcoded `background_agent` handoff tool + * whose call stays open for the whole backend turn, the session accepts an + * arbitrary tool list via config and dispatches every function call through + * `onFunctionCall`, expecting a prompt receipt-style output for each call via + * `submitFunctionOutput`. + */ +import { randomUUID } from 'node:crypto'; +import WebSocket from 'ws'; +import type { SocketLike } from './socket.js'; +import { deriveWebSocketBase } from './socket.js'; +import { escapeAnsiCtrlCodes } from './sanitize.js'; + +export type RealtimeCallEpoch = string | number; + +export const QWEN_REALTIME_INPUT_SAMPLE_RATE = 16_000; +export const QWEN_REALTIME_OUTPUT_SAMPLE_RATE = 24_000; + +export const QWEN_REALTIME_LIMITS = { + maxInputAudioFrameBytes: 64 * 1024, + maxOutputAudioFrameBytes: 256 * 1024, + maxBufferedSocketBytes: 1024 * 1024, + maxIncomingMessageBytes: 1024 * 1024, + maxTranscriptChars: 256 * 1024, + maxTextDeltaChars: 64 * 1024, + maxFunctionArgumentsChars: 32 * 1024, + maxFunctionOutputChars: 64 * 1024, + maxPendingFunctionCalls: 8, + maxIdentifierChars: 256, +} as const; + +const CONNECT_TIMEOUT_MS = 8000; +const MAX_ERROR_MESSAGE_CHARS = 300; +const MAX_RECENT_EVENT_IDS = 512; +const MAX_TRACKED_INPUT_ITEMS = 32; +const MAX_RETAINED_TRANSCRIPT_ENTRIES = 512; +export const REMAIN_SILENT_TOOL_NAME = 'remain_silent'; +const REALTIME_BACKEND_TEXT_PREFIX = '[BACKEND] '; +const REALTIME_SPEAK_TO_USER_PREFIX = '[SPEAK_TO_USER] '; + +/** + * OpenAI-style function tool declaration forwarded to the realtime provider. + * + * `capturesTranscript` marks handoff-style tools: when the model calls one, + * the session captures the live transcript tail (so the orchestrator can pack + * the recent voice context into the backend prompt) and marks the response as + * delegated, which keeps the same user turn out of the direct-answer + * transcript collection. + */ +export interface RealtimeToolDefinition { + type: 'function'; + function: { + name: string; + description: string; + parameters: Record; + }; + capturesTranscript?: boolean; +} +export interface QwenRealtimeConfig { + endpoint: string; + apiKey?: string; + model: string; + callEpoch: RealtimeCallEpoch; + voice?: string; + instructions: string; + tools: readonly RealtimeToolDefinition[]; +} + +export interface QwenRealtimeDeps { + createWebSocket?: ( + url: string, + options: { + headers: Record; + maxPayload: number; + perMessageDeflate: false; + handshakeTimeout: number; + }, + ) => SocketLike; + abortSignal?: AbortSignal; + connectTimeoutMs?: number; +} + +export interface RealtimeEventContext { + callEpoch: RealtimeCallEpoch; + eventId?: string; +} + +export interface RealtimeSpeechEvent extends RealtimeEventContext { + itemId?: string; + audioStartMs?: number; + audioEndMs?: number; +} + +export interface RealtimeInputTranscriptEvent extends RealtimeEventContext { + itemId?: string; + text: string; + stash?: string; + language?: string; + emotion?: string; +} + +export interface RealtimeResponseEvent extends RealtimeEventContext { + responseId: string; + inputItemId?: string; + status?: string; +} + +export type RealtimeResponseAuthority = 'direct' | 'backend_speech'; + +export interface RealtimeResponseCreatedEvent extends RealtimeResponseEvent { + authority: RealtimeResponseAuthority; +} + +export interface RealtimeOutputTextEvent extends RealtimeResponseEvent { + itemId?: string; + text: string; + source: 'text' | 'audio_transcript'; +} + +export interface RealtimeOutputAudioEvent extends RealtimeResponseEvent { + itemId?: string; + audio: Uint8Array; +} + +export interface RealtimeFunctionArgumentsEvent extends RealtimeResponseEvent { + itemId?: string; + callId: string; + delta: string; +} + +export interface RealtimeTranscriptEntry { + role: 'user' | 'assistant'; + text: string; +} + +export interface RealtimeDirectTranscriptEvent extends RealtimeEventContext { + entries: readonly RealtimeTranscriptEntry[]; +} + +export interface RealtimeFunctionCall extends RealtimeResponseEvent { + itemId?: string; + callId: string; + name: string; + /** Raw JSON argument string as sent by the model; parsing is the caller's job. */ + arguments: string; + /** Transcript tail captured at call time for `capturesTranscript` tools; empty otherwise. */ + activeTranscript: readonly RealtimeTranscriptEntry[]; +} + +export interface RealtimeIgnoredEvent extends RealtimeEventContext { + type: string; + reason: + | 'duplicate_event' + | 'stale_response' + | 'stale_input' + | 'stale_call' + | 'cancelled_response'; +} + +export interface RealtimeCloseInfo { + reason: 'client' | 'remote' | 'error'; + error?: QwenRealtimeError; +} + +export interface QwenRealtimeCallbacks { + onReady?: (event: RealtimeEventContext & { sessionId?: string }) => void; + onSpeechStarted?: (event: RealtimeSpeechEvent) => void; + onSpeechStopped?: (event: RealtimeSpeechEvent) => void; + onInputCommitted?: (event: RealtimeSpeechEvent) => void; + onInputTranscriptDelta?: (event: RealtimeInputTranscriptEvent) => void; + onInputTranscriptDone?: (event: RealtimeInputTranscriptEvent) => void; + onOutputTextDelta?: (event: RealtimeOutputTextEvent) => void; + onOutputTextDone?: (event: RealtimeOutputTextEvent) => void; + onOutputAudioDelta?: (event: RealtimeOutputAudioEvent) => void; + onOutputAudioDone?: ( + event: RealtimeResponseEvent & { itemId?: string }, + ) => void; + onFunctionArgumentsDelta?: (event: RealtimeFunctionArgumentsEvent) => void; + onFunctionCall?: (event: RealtimeFunctionCall) => void; + onResponseCreated?: (event: RealtimeResponseCreatedEvent) => void; + onResponseDone?: (event: RealtimeResponseEvent) => void; + onDirectTranscript?: (event: RealtimeDirectTranscriptEvent) => void; + onBargeIn?: (event: RealtimeResponseEvent) => void; + onIgnoredEvent?: (event: RealtimeIgnoredEvent) => void; + onAudioDropped?: (event: RealtimeEventContext) => void; + onError?: (error: QwenRealtimeError) => void; + onClose?: (info: RealtimeCloseInfo) => void; +} + +export interface RealtimeFunctionCallRef { + callEpoch: RealtimeCallEpoch; + callId: string; +} + +export interface RealtimeCloseOptions { + discardPendingInput?: boolean; +} + +export interface QwenRealtimeSession { + readonly callEpoch: RealtimeCallEpoch; + readonly closed: Promise; + pushAudio: (pcm16: Uint8Array) => boolean; + commitInputAudio: () => boolean; + clearInputAudio: () => boolean; + cancelResponse: () => boolean; + /** Submit the (receipt-style) output for a dispatched function call. */ + submitFunctionOutput: ( + ref: RealtimeFunctionCallRef, + output: string, + ) => boolean; + sendBackendContext: (text: string) => boolean; + speakToUser: (message: string) => boolean; + takeTranscriptTail: () => readonly RealtimeTranscriptEntry[]; + close: (options?: RealtimeCloseOptions) => void; +} + +export type QwenRealtimeErrorKind = 'configuration' | 'transient' | 'protocol'; + +export interface QwenRealtimeErrorOptions { + kind?: QwenRealtimeErrorKind; + status?: number; + providerType?: string; + param?: string; + closeCode?: number; +} + +function classifyRealtimeErrorKind( + code: string | undefined, + message: string, + status?: number, +): QwenRealtimeErrorKind { + const normalized = `${code ?? ''} ${message}`.toLowerCase(); + if ( + status === 429 || + (status !== undefined && status >= 500 && status <= 599) || + [ + 'connection_closed', + 'connection_failed', + 'connection_timeout', + 'socket_error', + 'send_failed', + ].includes(code ?? '') || + /\b429\b|rate[ _.-]?limit|throttl|limit_requests|limitrequests|resourceexhausted|resource_exhausted|limit_burst_rate|insufficient_quota|service_unavailable|internal[ _.-]?error|system[ _.-]?error|modelservicefailed|timeout|temporar/.test( + normalized, + ) + ) { + return 'transient'; + } + if ( + status === 400 || + status === 401 || + status === 403 || + status === 404 || + /invalid[ _.-]?(api[ _.-]?key|endpoint|model|request)|authentication|unauthori[sz]ed|forbidden|permission|model[ _.-]?(not[ _.-]?found|not[ _.-]?supported)/.test( + normalized, + ) + ) { + return 'configuration'; + } + return 'protocol'; +} + +export class QwenRealtimeError extends Error { + readonly code?: string; + readonly fatal: boolean; + readonly kind: QwenRealtimeErrorKind; + readonly status?: number; + readonly providerType?: string; + readonly param?: string; + readonly closeCode?: number; + + constructor( + message: string, + code?: string, + fatal = true, + options: QwenRealtimeErrorOptions = {}, + ) { + super(message); + this.name = 'QwenRealtimeError'; + this.code = code; + this.fatal = fatal; + this.kind = + options.kind ?? classifyRealtimeErrorKind(code, message, options.status); + this.status = options.status; + this.providerType = options.providerType; + this.param = options.param; + this.closeCode = options.closeCode; + } +} + +interface PendingFunctionCall { + responseId: string; + itemId?: string; + callId: string; + name?: string; + arguments: string; + dispatched: boolean; + outputSubmitted: boolean; + responseCompleted: boolean; + pendingOutput?: { + output: string; + }; +} + +interface ResponseCreateRequest { + authority: Exclude; + speechMessage?: string; + speechGeneration: number; + cancelled: boolean; +} + +interface ProviderMessage extends Record { + type?: unknown; + event_id?: unknown; +} + +export function deriveQwenOmniRealtimeUrl( + endpoint: string, + model: string, +): string { + const parsed = new URL(endpoint); + if (!['http:', 'https:', 'ws:', 'wss:'].includes(parsed.protocol)) { + throw new Error('Realtime endpoint must use HTTP or WebSocket.'); + } + if (parsed.username || parsed.password) { + throw new Error('Realtime endpoint must not contain credentials.'); + } + for (const name of parsed.searchParams.keys()) { + if (/api.?key|authorization|token/i.test(name)) { + throw new Error('Realtime endpoint must not contain credentials.'); + } + } + + let url: URL; + if (parsed.protocol === 'http:' || parsed.protocol === 'https:') { + const base = deriveWebSocketBase(parsed.toString()); + url = new URL( + parsed.pathname.replace(/\/+$/, '').endsWith('/api-ws/v1/realtime') + ? base + : `${base}/api-ws/v1/realtime`, + ); + for (const [name, value] of parsed.searchParams) { + url.searchParams.append(name, value); + } + } else { + url = parsed; + if (!url.pathname.replace(/\/+$/, '').endsWith('/api-ws/v1/realtime')) { + url.pathname = `${url.pathname.replace(/\/+$/, '')}/api-ws/v1/realtime`; + } + } + url.searchParams.set('model', model); + return url.toString(); +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function optionalString( + value: unknown, + maxChars: number = QWEN_REALTIME_LIMITS.maxIdentifierChars, +): string | undefined { + return typeof value === 'string' && value.length <= maxChars + ? value + : undefined; +} + +function optionalFiniteNumber(value: unknown): number | undefined { + return typeof value === 'number' && Number.isFinite(value) + ? value + : undefined; +} + +function sanitizeErrorText(raw: unknown, apiKey?: string): string { + let text = + typeof raw === 'string' + ? raw + : Buffer.isBuffer(raw) || raw instanceof Uint8Array + ? Buffer.from(raw).toString('utf8') + : 'Qwen Realtime request failed.'; + if (apiKey) text = text.split(apiKey).join('[REDACTED]'); + return escapeAnsiCtrlCodes(text).slice(0, MAX_ERROR_MESSAGE_CHARS); +} + +function optionalHttpStatus(value: unknown): number | undefined { + const status = + typeof value === 'string' && /^\d{3}$/.test(value) + ? Number(value) + : optionalFiniteNumber(value); + return status !== undefined && status >= 100 && status <= 599 + ? Math.trunc(status) + : undefined; +} + +function parseAudioDelta(value: unknown): Uint8Array | undefined { + if (typeof value !== 'string' || value.length === 0) return undefined; + const maxBase64Chars = + Math.ceil(QWEN_REALTIME_LIMITS.maxOutputAudioFrameBytes / 3) * 4 + 4; + if (value.length > maxBase64Chars || !/^[A-Za-z0-9+/]*={0,2}$/.test(value)) { + return undefined; + } + const decoded = Buffer.from(value, 'base64'); + if ( + decoded.length === 0 || + decoded.length % 2 !== 0 || + decoded.length > QWEN_REALTIME_LIMITS.maxOutputAudioFrameBytes + ) { + return undefined; + } + return new Uint8Array(decoded); +} + +export function openQwenRealtimeSession( + config: QwenRealtimeConfig, + callbacks: QwenRealtimeCallbacks = {}, + deps: QwenRealtimeDeps = {}, +): Promise { + const connectTimeoutMs = deps.connectTimeoutMs ?? CONNECT_TIMEOUT_MS; + const createWebSocket = + deps.createWebSocket ?? + ((url, options) => + new WebSocket(url, { + headers: options.headers, + maxPayload: options.maxPayload, + perMessageDeflate: options.perMessageDeflate, + handshakeTimeout: options.handshakeTimeout, + }) as unknown as SocketLike); + + return new Promise((resolve, reject) => { + if (deps.abortSignal?.aborted) { + reject(new QwenRealtimeError('Realtime connection was aborted.')); + return; + } + + let realtimeUrl: string; + try { + realtimeUrl = deriveQwenOmniRealtimeUrl(config.endpoint, config.model); + } catch (error) { + reject( + new QwenRealtimeError( + sanitizeErrorText( + error instanceof Error ? error.message : error, + config.apiKey, + ), + 'invalid_endpoint', + true, + { kind: 'configuration' }, + ), + ); + return; + } + + let ws: SocketLike; + try { + ws = createWebSocket(realtimeUrl, { + headers: config.apiKey + ? { Authorization: `Bearer ${config.apiKey}` } + : {}, + maxPayload: QWEN_REALTIME_LIMITS.maxIncomingMessageBytes, + perMessageDeflate: false, + handshakeTimeout: connectTimeoutMs, + }); + } catch (error) { + reject( + new QwenRealtimeError( + sanitizeErrorText( + error instanceof Error ? error.message : error, + config.apiKey, + ), + 'connection_failed', + true, + { kind: 'transient' }, + ), + ); + return; + } + + let ready = false; + let settled = false; + let terminal = false; + let closedByClient = false; + let sessionUpdateSent = false; + let activeResponseId: string | undefined; + let lastCompletedResponseId: string | undefined; + let activeAudioResponseId: string | undefined; + let pendingResponseCreate: ResponseCreateRequest | undefined; + let responseCreateQueue: ResponseCreateRequest[] = []; + let speechGeneration = 0; + let activeResponseAuthority: RealtimeResponseAuthority | undefined; + const toolsByName = new Map( + config.tools.map((tool) => [tool.function.name, tool]), + ); + let backpressureWarned = false; + let speechInputInProgress = false; + let speechCommitPending = false; + let connectTimer: ReturnType | undefined; + let abortListener: (() => void) | undefined; + const cancelledResponseIds = new Set(); + const recentEventIds = new Set(); + const pendingCalls = new Map(); + const committedInputItemIds = new Set(); + const completedInputTranscripts = new Map(); + const responseInputItemIds = new Map(); + const consumedInputItemIds = new Set(); + const transcriptEntries: RealtimeTranscriptEntry[] = []; + const pendingDirectTranscriptEntries: RealtimeTranscriptEntry[] = []; + const collectedDirectResponseIds = new Set(); + const collectedDirectInputItemIds = new Set(); + const responseAuthorities = new Map(); + const delegatedResponseIds = new Set(); + const responseOutputText = new Map< + string, + { text: string; audioTranscript: string } + >(); + let newInputEntry = false; + let newOutputEntry = false; + let resolveClosed: (info: RealtimeCloseInfo) => void = () => undefined; + const closed = new Promise((res) => { + resolveClosed = res; + }); + let closedSettled = false; + + const callback = (fn: (() => void) | undefined): boolean => { + if (!fn) return true; + try { + fn(); + return true; + } catch (error) { + fail( + new QwenRealtimeError( + sanitizeErrorText( + error instanceof Error ? error.message : error, + config.apiKey, + ), + 'callback_failed', + ), + ); + return false; + } + }; + + const settleClosed = (info: RealtimeCloseInfo) => { + if (closedSettled) return; + closedSettled = true; + resolveClosed(info); + try { + callbacks.onClose?.(info); + } catch { + /* ignore observer failures after shutdown */ + } + }; + + const clearConnectTimer = () => { + if (!connectTimer) return; + clearTimeout(connectTimer); + connectTimer = undefined; + }; + + const removeAbortListener = () => { + if (!abortListener) return; + deps.abortSignal?.removeEventListener('abort', abortListener); + abortListener = undefined; + }; + + const closeSocket = () => { + try { + ws.close(); + } catch { + /* ignore */ + } + }; + + const notifyError = (error: QwenRealtimeError) => { + try { + callbacks.onError?.(error); + } catch { + /* ignore error observer failures */ + } + }; + + const unrecoverableInputError = (): QwenRealtimeError => + new QwenRealtimeError( + 'Realtime connection ended before accepted speech finished transcribing.', + 'unrecoverable_input', + true, + { kind: 'protocol' }, + ); + + const pendingInputLossError = (): QwenRealtimeError | undefined => { + const hasUnresolvedCommittedInput = [...committedInputItemIds].some( + (itemId) => !completedInputTranscripts.has(itemId), + ); + return speechInputInProgress || + speechCommitPending || + hasUnresolvedCommittedInput + ? unrecoverableInputError() + : undefined; + }; + + function fail(error: QwenRealtimeError): void { + if (terminal) return; + const inputLossError = pendingInputLossError(); + const reportedError = + error.kind !== 'protocol' && inputLossError ? inputLossError : error; + terminal = true; + clearConnectTimer(); + removeAbortListener(); + closeSocket(); + if (!settled) { + settled = true; + reject(reportedError); + } else { + notifyError(reportedError); + } + settleClosed({ reason: 'error', error: reportedError }); + } + + const protocolError = (message: string, code: string) => { + fail(new QwenRealtimeError(message, code)); + }; + + const sendJson = (body: Record): boolean => { + if (terminal || closedByClient || ws.readyState !== ws.OPEN) return false; + try { + ws.send(JSON.stringify({ event_id: randomUUID(), ...body })); + return true; + } catch (error) { + fail( + new QwenRealtimeError( + sanitizeErrorText( + error instanceof Error ? error.message : error, + config.apiKey, + ), + 'send_failed', + ), + ); + return false; + } + }; + + const markResponseCancelled = (responseId: string): void => { + if (cancelledResponseIds.has(responseId)) return; + cancelledResponseIds.add(responseId); + for (const [callId, call] of pendingCalls) { + if (call.responseId === responseId && !call.dispatched) { + pendingCalls.delete(callId); + } + } + if (activeResponseId === responseId) { + activeResponseId = undefined; + activeResponseAuthority = undefined; + } + if (activeAudioResponseId === responseId) { + activeAudioResponseId = undefined; + } + if (cancelledResponseIds.size > 16) { + const oldest = cancelledResponseIds.values().next().value; + if (typeof oldest === 'string') cancelledResponseIds.delete(oldest); + } + }; + + const sendFunctionCallOutput = ( + call: PendingFunctionCall, + output: string, + ): boolean => { + if ( + !sendJson({ + type: 'conversation.item.create', + item: { + type: 'function_call_output', + call_id: call.callId, + output, + }, + }) + ) { + return false; + } + call.outputSubmitted = true; + call.pendingOutput = undefined; + pendingCalls.delete(call.callId); + return true; + }; + + const sendResponseCreate = (request: ResponseCreateRequest): boolean => { + if ( + request.authority === 'backend_speech' && + request.speechGeneration !== speechGeneration + ) { + return true; + } + if ( + request.speechMessage !== undefined && + !sendBackendConversationItem( + request.speechMessage, + REALTIME_SPEAK_TO_USER_PREFIX, + ) + ) { + return false; + } + pendingResponseCreate = request; + if ( + sendJson({ + type: 'response.create', + ...(request.authority === 'backend_speech' + ? { response: { modalities: ['text', 'audio'] } } + : {}), + }) + ) { + return true; + } + pendingResponseCreate = undefined; + return false; + }; + + const requestResponseCreate = ( + authority: Exclude, + speechMessage?: string, + ): boolean => { + const request = { + authority, + ...(speechMessage !== undefined ? { speechMessage } : {}), + speechGeneration, + cancelled: false, + }; + if (pendingResponseCreate || activeResponseId) { + responseCreateQueue.push(request); + return true; + } + return sendResponseCreate(request); + }; + + const flushResponseCreate = (): void => { + if (pendingResponseCreate || activeResponseId) { + return; + } + const next = responseCreateQueue.shift(); + if (!next) return; + if (next.cancelled) { + queueMicrotask(flushResponseCreate); + return; + } + sendResponseCreate(next); + }; + + const sendBackendConversationItem = ( + text: string, + prefix = REALTIME_BACKEND_TEXT_PREFIX, + ): boolean => + sendJson({ + type: 'conversation.item.create', + item: { + type: 'message', + role: 'user', + content: [ + { + type: 'input_text', + text: `${prefix}${text}`, + }, + ], + }, + }); + + const queueFunctionCallOutput = ( + call: PendingFunctionCall, + output: string, + ): boolean => { + if (call.outputSubmitted || call.pendingOutput) return false; + if (!call.responseCompleted && activeResponseId === call.responseId) { + call.pendingOutput = { output }; + return true; + } + if (!sendFunctionCallOutput(call, output)) return false; + return true; + }; + + const completePendingCallsForResponse = ( + responseId: string, + status: string | undefined, + ): void => { + for (const [callId, call] of [...pendingCalls]) { + if (call.responseId !== responseId) continue; + call.responseCompleted = true; + if (status === 'failed' || !call.dispatched) { + pendingCalls.delete(callId); + continue; + } + const pendingOutput = call.pendingOutput; + if (!pendingOutput) continue; + call.pendingOutput = undefined; + sendFunctionCallOutput(call, pendingOutput.output); + } + }; + + const rememberCompletedInputTranscript = ( + itemId: string, + transcript: string, + ): boolean => { + if (consumedInputItemIds.has(itemId)) return false; + const existing = completedInputTranscripts.get(itemId); + if (existing !== undefined) { + protocolError( + 'Realtime provider supplied multiple final transcripts for one input.', + 'ambiguous_final_transcript', + ); + return false; + } + if ( + completedInputTranscripts.size >= MAX_TRACKED_INPUT_ITEMS && + !completedInputTranscripts.has(itemId) + ) { + protocolError( + 'Realtime provider created too many pending final transcripts.', + 'too_many_pending_inputs', + ); + return false; + } + completedInputTranscripts.set(itemId, transcript); + return true; + }; + + const consumeInputItem = (itemId: string): void => { + completedInputTranscripts.delete(itemId); + committedInputItemIds.delete(itemId); + collectedDirectInputItemIds.delete(itemId); + consumedInputItemIds.add(itemId); + while (consumedInputItemIds.size > MAX_TRACKED_INPUT_ITEMS) { + const oldest = consumedInputItemIds.values().next().value; + if (typeof oldest !== 'string') break; + consumedInputItemIds.delete(oldest); + } + }; + + const consumeResponseInput = (responseId: string): string | undefined => { + const itemId = responseInputItemIds.get(responseId); + responseInputItemIds.delete(responseId); + if (itemId) consumeInputItem(itemId); + return itemId; + }; + + const pushTranscriptEntry = (entry: RealtimeTranscriptEntry): void => { + transcriptEntries.push(entry); + // Long stretches without a capturing handoff must not grow the + // retained transcript without bound: evict the oldest entries past a + // fixed cap. + while (transcriptEntries.length > MAX_RETAINED_TRANSCRIPT_ENTRIES) { + transcriptEntries.shift(); + } + }; + + const appendTranscriptDelta = ( + role: RealtimeTranscriptEntry['role'], + delta: string, + forceNew: boolean, + ): void => { + if (!delta) return; + const last = transcriptEntries.at(-1); + if (!forceNew && last?.role === role) { + last.text += delta; + return; + } + pushTranscriptEntry({ role, text: delta }); + }; + + const applyTranscriptDone = ( + role: RealtimeTranscriptEntry['role'], + text: string, + forceNew: boolean, + ): void => { + if (!text) return; + const last = transcriptEntries.at(-1); + if (!forceNew && last?.role === role) { + last.text = text; + return; + } + pushTranscriptEntry({ role, text }); + }; + + const deliverDirectTranscript = ( + entries: readonly RealtimeTranscriptEntry[], + ): void => { + if (entries.length === 0) return; + const copiedEntries = entries.map((entry) => ({ ...entry })); + if (!callbacks.onDirectTranscript) { + pendingDirectTranscriptEntries.push(...copiedEntries); + return; + } + if ( + !callback(() => + callbacks.onDirectTranscript?.({ + callEpoch: config.callEpoch, + entries: copiedEntries, + }), + ) + ) { + pendingDirectTranscriptEntries.push(...copiedEntries); + } + }; + + const collectDirectTranscript = (responseId: string): void => { + if ( + collectedDirectResponseIds.has(responseId) || + responseAuthorities.get(responseId) !== 'direct' || + delegatedResponseIds.has(responseId) + ) { + return; + } + const inputItemId = responseInputItemIds.get(responseId); + const input = inputItemId + ? completedInputTranscripts.get(inputItemId) + : undefined; + const output = responseOutputText.get(responseId); + const assistant = output?.audioTranscript || output?.text; + const entries: RealtimeTranscriptEntry[] = []; + if (inputItemId && input?.trim()) { + collectedDirectInputItemIds.add(inputItemId); + entries.push({ role: 'user', text: input }); + } + if (assistant?.trim()) { + entries.push({ role: 'assistant', text: assistant }); + } + if (entries.length === 0) return; + collectedDirectResponseIds.add(responseId); + deliverDirectTranscript(entries); + }; + + const takeTranscriptTail = (): readonly RealtimeTranscriptEntry[] => { + for (const responseId of responseAuthorities.keys()) { + collectDirectTranscript(responseId); + } + const delegatedInputItemIds = new Set( + [...delegatedResponseIds] + .map((responseId) => responseInputItemIds.get(responseId)) + .filter((itemId): itemId is string => itemId !== undefined), + ); + for (const [itemId, text] of completedInputTranscripts) { + if ( + collectedDirectInputItemIds.has(itemId) || + delegatedInputItemIds.has(itemId) || + !text.trim() + ) { + continue; + } + collectedDirectInputItemIds.add(itemId); + deliverDirectTranscript([{ role: 'user', text }]); + } + return pendingDirectTranscriptEntries.splice(0).map((entry) => ({ + ...entry, + })); + }; + + const takeHandoffTranscriptTail = + (): readonly RealtimeTranscriptEntry[] => { + const tail = transcriptEntries.map((entry) => ({ ...entry })); + // Everything up to here has now been handed off and is never read + // again; drop the consumed entries (instead of only advancing a + // cursor) so retention and the dedup scan in takeHandoffTranscript + // stay bounded by one capture window, not the whole call history. + transcriptEntries.length = 0; + return tail; + }; + + const updateResponseOutputText = ( + responseId: string, + source: RealtimeOutputTextEvent['source'], + text: string, + done: boolean, + ): void => { + const current = responseOutputText.get(responseId) ?? { + text: '', + audioTranscript: '', + }; + const key = source === 'audio_transcript' ? 'audioTranscript' : 'text'; + current[key] = done ? text : `${current[key]}${text}`; + responseOutputText.set(responseId, current); + }; + + const takeHandoffTranscript = ( + input: string, + ): readonly RealtimeTranscriptEntry[] => { + const trimmed = input.trim(); + if ( + trimmed && + !transcriptEntries.some( + (entry) => entry.role === 'user' && entry.text.trim() === trimmed, + ) + ) { + pushTranscriptEntry({ role: 'user', text: trimmed }); + } + const transcript = takeHandoffTranscriptTail(); + newInputEntry = true; + newOutputEntry = true; + return transcript; + }; + + const bindResponseInput = (responseId: string): boolean => { + if (responseInputItemIds.has(responseId)) return true; + const boundInputItemIds = new Set(responseInputItemIds.values()); + const candidates = [...committedInputItemIds].filter( + (itemId) => + !boundInputItemIds.has(itemId) && !consumedInputItemIds.has(itemId), + ); + if (candidates.length > 1) { + protocolError( + 'Realtime response could not be associated with one unique input.', + 'ambiguous_input_transcript', + ); + return false; + } + const itemId = candidates[0]; + if (!itemId) return false; + responseInputItemIds.set(responseId, itemId); + return true; + }; + + const finalizeCancelledResponse = (responseId: string): void => { + if (!cancelledResponseIds.has(responseId)) { + markResponseCancelled(responseId); + } + const responseInputItemId = responseInputItemIds.get(responseId); + collectDirectTranscript(responseId); + completePendingCallsForResponse(responseId, 'cancelled'); + consumeResponseInput(responseId); + if (activeResponseId === responseId) { + activeResponseId = undefined; + activeResponseAuthority = undefined; + } + if (activeAudioResponseId === responseId) { + activeAudioResponseId = undefined; + } + lastCompletedResponseId = responseId; + cancelledResponseIds.delete(responseId); + callback(() => + callbacks.onResponseDone?.({ + callEpoch: config.callEpoch, + responseId, + ...(responseInputItemId ? { inputItemId: responseInputItemId } : {}), + status: 'cancelled', + }), + ); + responseAuthorities.delete(responseId); + delegatedResponseIds.delete(responseId); + responseOutputText.delete(responseId); + collectedDirectResponseIds.delete(responseId); + queueMicrotask(flushResponseCreate); + }; + + const eventContext = (message: ProviderMessage): RealtimeEventContext => ({ + callEpoch: config.callEpoch, + eventId: optionalString(message.event_id), + }); + + const ignoreEvent = ( + message: ProviderMessage, + type: string, + reason: RealtimeIgnoredEvent['reason'], + ) => { + callback(() => + callbacks.onIgnoredEvent?.({ + ...eventContext(message), + type, + reason, + }), + ); + }; + + const readResponseId = ( + message: ProviderMessage, + fromResponseObject = false, + ): string | undefined => { + const response = isRecord(message['response']) + ? message['response'] + : undefined; + return optionalString( + fromResponseObject ? response?.['id'] : message['response_id'], + ); + }; + + const isCurrentResponse = ( + message: ProviderMessage, + type: string, + responseId: string, + ): boolean => { + if (cancelledResponseIds.has(responseId)) { + ignoreEvent(message, type, 'cancelled_response'); + return false; + } + if (activeResponseId !== responseId) { + ignoreEvent(message, type, 'stale_response'); + return false; + } + return true; + }; + + /** + * Best-effort extraction of the user-request text from a tool-call + * argument payload. Only used to augment the captured transcript for + * `capturesTranscript` tools (ASR may lag behind the model's call); the + * raw argument string is always handed to the orchestrator untouched. + */ + const extractInputTranscript = (rawArguments: string): string => { + if (rawArguments.length === 0) return ''; + try { + const parsed = JSON.parse(rawArguments) as unknown; + if (isRecord(parsed)) { + for (const key of [ + 'task', + 'input_transcript', + 'input', + 'text', + 'prompt', + 'query', + ]) { + const value = parsed[key]; + if (typeof value === 'string' && value.trim()) { + return value.trim(); + } + } + } + } catch { + /* match the Codex V2 parser: use raw arguments when JSON is invalid */ + } + return ''; + }; + + const dispatchFunctionCall = ( + message: ProviderMessage, + call: PendingFunctionCall, + rawArguments: string, + ): void => { + if (call.dispatched) { + if (call.arguments !== rawArguments) { + protocolError( + 'Realtime model changed a function call after dispatch.', + 'ambiguous_handoff', + ); + } + return; + } + if (call.name === REMAIN_SILENT_TOOL_NAME) { + call.arguments = rawArguments; + call.dispatched = true; + queueFunctionCallOutput(call, ''); + return; + } + const tool = call.name ? toolsByName.get(call.name) : undefined; + if (!tool) { + // A tool the config never declared: answer with an error receipt so + // the model can recover aloud instead of waiting on a call that no + // handler will ever complete. + call.arguments = rawArguments; + call.dispatched = true; + queueFunctionCallOutput( + call, + JSON.stringify({ + status: 'error', + note: `Unknown tool: ${call.name ?? '(unnamed)'}`, + }), + ); + return; + } + call.arguments = rawArguments; + call.dispatched = true; + let activeTranscript: readonly RealtimeTranscriptEntry[] = []; + if (tool.capturesTranscript) { + activeTranscript = takeHandoffTranscript( + extractInputTranscript(rawArguments), + ); + delegatedResponseIds.add(call.responseId); + } + callback(() => + callbacks.onFunctionCall?.({ + ...eventContext(message), + responseId: call.responseId, + inputItemId: responseInputItemIds.get(call.responseId), + itemId: call.itemId, + callId: call.callId, + name: call.name ?? '', + arguments: rawArguments, + activeTranscript, + }), + ); + }; + + const session: QwenRealtimeSession = { + callEpoch: config.callEpoch, + closed, + pushAudio: (pcm16) => { + if (pcm16.length === 0) return false; + if ( + pcm16.length % 2 !== 0 || + pcm16.length > QWEN_REALTIME_LIMITS.maxInputAudioFrameBytes + ) { + throw new RangeError( + 'Realtime input must be a bounded PCM16 audio frame.', + ); + } + if ( + terminal || + closedByClient || + ws.readyState !== ws.OPEN || + (ws.bufferedAmount ?? 0) > QWEN_REALTIME_LIMITS.maxBufferedSocketBytes + ) { + if (!backpressureWarned) { + backpressureWarned = true; + callback(() => + callbacks.onAudioDropped?.({ callEpoch: config.callEpoch }), + ); + } + return false; + } + backpressureWarned = false; + return sendJson({ + type: 'input_audio_buffer.append', + audio: Buffer.from(pcm16).toString('base64'), + }); + }, + commitInputAudio: () => sendJson({ type: 'input_audio_buffer.commit' }), + clearInputAudio: () => { + const sent = sendJson({ type: 'input_audio_buffer.clear' }); + if (sent) { + speechInputInProgress = false; + speechCommitPending = false; + } + return sent; + }, + cancelResponse: () => { + if (!activeResponseId || cancelledResponseIds.has(activeResponseId)) { + return false; + } + const responseId = activeResponseId; + markResponseCancelled(responseId); + const sent = sendJson({ type: 'response.cancel' }); + finalizeCancelledResponse(responseId); + return sent; + }, + submitFunctionOutput: (ref, output) => { + const call = pendingCalls.get(ref.callId); + if ( + ref.callEpoch !== config.callEpoch || + !call || + !call.dispatched || + call.outputSubmitted || + terminal || + closedByClient + ) { + callback(() => + callbacks.onIgnoredEvent?.({ + callEpoch: config.callEpoch, + type: 'conversation.item.create', + reason: 'stale_call', + }), + ); + return false; + } + if ( + typeof output !== 'string' || + output.trim().length === 0 || + output.length > QWEN_REALTIME_LIMITS.maxFunctionOutputChars + ) { + throw new RangeError( + 'Realtime function output exceeded the allowed size.', + ); + } + return queueFunctionCallOutput(call, output); + }, + sendBackendContext: (text) => { + if ( + typeof text !== 'string' || + text.trim().length === 0 || + text.length > QWEN_REALTIME_LIMITS.maxFunctionOutputChars + ) { + throw new RangeError( + 'Realtime backend context exceeded the allowed size.', + ); + } + if (terminal || closedByClient) return false; + return sendBackendConversationItem(text); + }, + speakToUser: (message) => { + if ( + typeof message !== 'string' || + message.trim().length === 0 || + message.length > QWEN_REALTIME_LIMITS.maxFunctionOutputChars + ) { + throw new RangeError( + 'Realtime speech request exceeded the allowed size.', + ); + } + if (terminal || closedByClient) return false; + return requestResponseCreate('backend_speech', message); + }, + takeTranscriptTail, + close: (options) => { + if (closedByClient || terminal) return; + if (!options?.discardPendingInput) { + const inputLossError = pendingInputLossError(); + if (inputLossError) { + fail(inputLossError); + return; + } + } + closedByClient = true; + clearConnectTimer(); + removeAbortListener(); + closeSocket(); + settleClosed({ reason: 'client' }); + }, + }; + + const sendSessionUpdate = () => { + if (sessionUpdateSent) return; + sessionUpdateSent = true; + sendJson({ + type: 'session.update', + session: { + modalities: ['text', 'audio'], + ...(config.voice ? { voice: config.voice } : {}), + input_audio_format: 'pcm', + output_audio_format: 'pcm', + input_audio_transcription: { + model: 'qwen3-asr-flash-realtime', + }, + instructions: config.instructions, + turn_detection: { + type: 'semantic_vad', + create_response: true, + interrupt_response: true, + }, + // Strip the local-only `capturesTranscript` flag from the wire shape. + tools: config.tools.map((tool) => ({ + type: tool.type, + function: tool.function, + })), + tool_choice: 'auto', + }, + }); + }; + + ws.on('message', (...args: unknown[]) => { + if (terminal || closedByClient) return; + if (args[1] === true) { + protocolError( + 'Realtime provider sent an unexpected binary message.', + 'unexpected_binary_message', + ); + return; + } + const raw = String(args[0]); + if ( + Buffer.byteLength(raw) > QWEN_REALTIME_LIMITS.maxIncomingMessageBytes + ) { + protocolError( + 'Realtime provider message exceeded the allowed size.', + 'message_too_large', + ); + return; + } + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch { + protocolError( + 'Realtime provider sent invalid JSON.', + 'invalid_provider_message', + ); + return; + } + if (!isRecord(parsed)) { + protocolError( + 'Realtime provider message must be an object.', + 'invalid_provider_message', + ); + return; + } + const message = parsed as ProviderMessage; + const type = optionalString(message.type); + if (!type) { + protocolError( + 'Realtime provider message omitted its type.', + 'invalid_provider_message', + ); + return; + } + const eventId = optionalString(message.event_id); + if (eventId) { + if (recentEventIds.has(eventId)) { + ignoreEvent(message, type, 'duplicate_event'); + return; + } + recentEventIds.add(eventId); + if (recentEventIds.size > MAX_RECENT_EVENT_IDS) { + const oldest = recentEventIds.values().next().value; + if (typeof oldest === 'string') recentEventIds.delete(oldest); + } + } + + switch (type) { + case 'session.created': { + sendSessionUpdate(); + break; + } + case 'session.updated': { + if (ready) break; + ready = true; + clearConnectTimer(); + const providerSession = isRecord(message['session']) + ? message['session'] + : undefined; + if ( + !callback(() => + callbacks.onReady?.({ + ...eventContext(message), + sessionId: optionalString(providerSession?.['id']), + }), + ) + ) { + break; + } + settled = true; + resolve(session); + break; + } + case 'input_audio_buffer.speech_started': { + const itemId = optionalString(message['item_id']); + newInputEntry = true; + speechInputInProgress = true; + speechCommitPending = true; + speechGeneration += 1; + responseCreateQueue = responseCreateQueue.filter( + (request) => request.authority !== 'backend_speech', + ); + if (pendingResponseCreate?.authority === 'backend_speech') { + pendingResponseCreate.cancelled = true; + } + if ( + activeResponseId && + (activeAudioResponseId === activeResponseId || + activeResponseAuthority === 'backend_speech') && + !cancelledResponseIds.has(activeResponseId) + ) { + const interruptedResponseId = activeResponseId; + callback(() => + callbacks.onBargeIn?.({ + ...eventContext(message), + responseId: interruptedResponseId, + }), + ); + markResponseCancelled(interruptedResponseId); + finalizeCancelledResponse(interruptedResponseId); + } + callback(() => + callbacks.onSpeechStarted?.({ + ...eventContext(message), + itemId, + audioStartMs: optionalFiniteNumber(message['audio_start_ms']), + }), + ); + break; + } + case 'input_audio_buffer.speech_stopped': { + const itemId = optionalString(message['item_id']); + speechInputInProgress = false; + callback(() => + callbacks.onSpeechStopped?.({ + ...eventContext(message), + itemId, + audioEndMs: optionalFiniteNumber(message['audio_end_ms']), + }), + ); + break; + } + case 'input_audio_buffer.committed': { + const itemId = optionalString(message['item_id']); + if (!itemId) { + protocolError( + 'Realtime committed input omitted its identifier.', + 'invalid_input_item', + ); + break; + } + if ( + committedInputItemIds.has(itemId) || + consumedInputItemIds.has(itemId) + ) { + protocolError( + 'Realtime provider reused a committed input identifier.', + 'ambiguous_input_transcript', + ); + break; + } + if (committedInputItemIds.size >= MAX_TRACKED_INPUT_ITEMS) { + protocolError( + 'Realtime provider created too many pending input items.', + 'too_many_pending_inputs', + ); + break; + } + speechInputInProgress = false; + speechCommitPending = false; + lastCompletedResponseId = undefined; + committedInputItemIds.add(itemId); + if ( + activeResponseId && + activeResponseAuthority === 'direct' && + !bindResponseInput(activeResponseId) + ) { + break; + } + callback(() => + callbacks.onInputCommitted?.({ + ...eventContext(message), + itemId, + }), + ); + break; + } + case 'conversation.item.input_audio_transcription.delta': + case 'conversation.item.input_audio_transcription.text': { + const itemId = optionalString(message['item_id']); + const text = optionalString( + message['text'] ?? message['delta'] ?? '', + QWEN_REALTIME_LIMITS.maxTranscriptChars, + ); + const stash = optionalString( + message['stash'] ?? '', + QWEN_REALTIME_LIMITS.maxTranscriptChars, + ); + if ( + text === undefined || + stash === undefined || + text.length + stash.length > QWEN_REALTIME_LIMITS.maxTranscriptChars + ) { + protocolError( + 'Realtime input transcript exceeded the allowed size.', + 'transcript_too_large', + ); + break; + } + const transcriptText = `${text}${stash}`; + if (type === 'conversation.item.input_audio_transcription.delta') { + appendTranscriptDelta('user', transcriptText, newInputEntry); + } else { + applyTranscriptDone('user', transcriptText, newInputEntry); + } + newInputEntry = false; + callback(() => + callbacks.onInputTranscriptDelta?.({ + ...eventContext(message), + itemId, + text: `${text}${stash}`, + stash, + language: optionalString(message['language']), + emotion: optionalString(message['emotion']), + }), + ); + break; + } + case 'conversation.item.input_audio_transcription.completed': { + const itemId = optionalString(message['item_id']); + if (itemId && consumedInputItemIds.has(itemId)) { + // A benign late final: barge-in or response.done already consumed + // this input before the ASR stream delivered its transcript. Drop + // it instead of treating a healthy call as a protocol violation. + ignoreEvent(message, type, 'stale_input'); + break; + } + if (!itemId || !committedInputItemIds.has(itemId)) { + protocolError( + 'Realtime final transcript had no committed input item.', + 'unattributed_final_transcript', + ); + break; + } + const transcript = optionalString( + message['transcript'], + QWEN_REALTIME_LIMITS.maxTranscriptChars, + ); + if (transcript === undefined) { + protocolError( + 'Realtime input transcript exceeded the allowed size.', + 'transcript_too_large', + ); + break; + } + if (!rememberCompletedInputTranscript(itemId, transcript)) break; + applyTranscriptDone('user', transcript, newInputEntry); + newInputEntry = false; + callback(() => + callbacks.onInputTranscriptDone?.({ + ...eventContext(message), + itemId, + text: transcript, + }), + ); + break; + } + case 'conversation.item.input_audio_transcription.failed': { + const error = isRecord(message['error']) + ? message['error'] + : undefined; + const inputLossError = pendingInputLossError(); + if (inputLossError) { + fail(inputLossError); + } else { + notifyError( + new QwenRealtimeError( + sanitizeErrorText( + error?.['message'] ?? + error?.['code'] ?? + 'Realtime input transcription failed.', + config.apiKey, + ), + optionalString(error?.['code']), + false, + ), + ); + } + break; + } + case 'response.created': { + const responseId = readResponseId(message, true); + if (!responseId) { + protocolError( + 'Realtime response omitted its identifier.', + 'invalid_response', + ); + break; + } + if (activeResponseId && activeResponseId !== responseId) { + const supersededResponseId = activeResponseId; + markResponseCancelled(supersededResponseId); + finalizeCancelledResponse(supersededResponseId); + } + const responseRequest = pendingResponseCreate; + const responseAuthority: RealtimeResponseAuthority = + responseRequest?.authority ?? 'direct'; + activeResponseId = responseId; + activeResponseAuthority = responseAuthority; + responseAuthorities.set(responseId, responseAuthority); + newOutputEntry = true; + if (activeResponseAuthority === 'direct') { + bindResponseInput(responseId); + if (terminal) break; + } + pendingResponseCreate = undefined; + activeAudioResponseId = undefined; + if (responseRequest?.cancelled) { + markResponseCancelled(responseId); + sendJson({ type: 'response.cancel' }); + finalizeCancelledResponse(responseId); + break; + } + const response = isRecord(message['response']) + ? message['response'] + : undefined; + const responseInputItemId = responseInputItemIds.get(responseId); + callback(() => + callbacks.onResponseCreated?.({ + ...eventContext(message), + responseId, + ...(responseInputItemId + ? { inputItemId: responseInputItemId } + : {}), + authority: responseAuthority, + status: optionalString(response?.['status']), + }), + ); + break; + } + case 'response.audio.delta': + case 'response.output_audio.delta': { + const responseId = readResponseId(message); + if (!responseId || !isCurrentResponse(message, type, responseId)) { + break; + } + const audio = parseAudioDelta(message['delta']); + if (!audio) { + protocolError( + 'Realtime output audio frame was invalid or too large.', + 'invalid_audio_frame', + ); + break; + } + activeAudioResponseId = responseId; + callback(() => + callbacks.onOutputAudioDelta?.({ + ...eventContext(message), + responseId, + itemId: optionalString(message['item_id']), + audio, + }), + ); + break; + } + case 'response.audio.done': + case 'response.output_audio.done': { + const responseId = readResponseId(message); + if (!responseId || !isCurrentResponse(message, type, responseId)) { + break; + } + if (activeAudioResponseId === responseId) { + activeAudioResponseId = undefined; + } + callback(() => + callbacks.onOutputAudioDone?.({ + ...eventContext(message), + responseId, + itemId: optionalString(message['item_id']), + }), + ); + break; + } + case 'response.text.delta': + case 'response.output_text.delta': + case 'response.audio_transcript.delta': { + const responseId = readResponseId(message); + if (!responseId || !isCurrentResponse(message, type, responseId)) { + break; + } + const delta = optionalString( + message['delta'], + QWEN_REALTIME_LIMITS.maxTextDeltaChars, + ); + if (delta === undefined) { + protocolError( + 'Realtime output text delta exceeded the allowed size.', + 'text_delta_too_large', + ); + break; + } + appendTranscriptDelta('assistant', delta, newOutputEntry); + newOutputEntry = false; + updateResponseOutputText( + responseId, + type.includes('audio_transcript') ? 'audio_transcript' : 'text', + delta, + false, + ); + callback(() => + callbacks.onOutputTextDelta?.({ + ...eventContext(message), + responseId, + itemId: optionalString(message['item_id']), + text: delta, + source: type.includes('audio_transcript') + ? 'audio_transcript' + : 'text', + }), + ); + break; + } + case 'response.text.done': + case 'response.output_text.done': + case 'response.audio_transcript.done': { + const responseId = readResponseId(message); + if (!responseId || !isCurrentResponse(message, type, responseId)) { + break; + } + const text = optionalString( + message['text'] ?? message['transcript'], + QWEN_REALTIME_LIMITS.maxTranscriptChars, + ); + if (text === undefined) { + protocolError( + 'Realtime output text exceeded the allowed size.', + 'transcript_too_large', + ); + break; + } + applyTranscriptDone('assistant', text, newOutputEntry); + newOutputEntry = false; + updateResponseOutputText( + responseId, + type.includes('audio_transcript') ? 'audio_transcript' : 'text', + text, + true, + ); + callback(() => + callbacks.onOutputTextDone?.({ + ...eventContext(message), + responseId, + itemId: optionalString(message['item_id']), + text, + source: type.includes('audio_transcript') + ? 'audio_transcript' + : 'text', + }), + ); + break; + } + case 'response.function_call_arguments.delta': { + const responseId = readResponseId(message); + if (!responseId || !isCurrentResponse(message, type, responseId)) { + break; + } + const callId = optionalString(message['call_id']); + const delta = optionalString( + message['delta'], + QWEN_REALTIME_LIMITS.maxFunctionArgumentsChars, + ); + if (!callId || delta === undefined) { + protocolError( + 'Realtime function argument event was invalid.', + 'invalid_function_arguments', + ); + break; + } + const existing = pendingCalls.get(callId); + if (existing && existing.responseId !== responseId) { + ignoreEvent(message, type, 'stale_call'); + break; + } + if (existing?.dispatched) { + protocolError( + 'Realtime model changed a handoff call after dispatch.', + 'ambiguous_handoff', + ); + break; + } + if ( + !existing && + pendingCalls.size >= QWEN_REALTIME_LIMITS.maxPendingFunctionCalls + ) { + protocolError( + 'Realtime provider created too many pending function calls.', + 'too_many_function_calls', + ); + break; + } + const call: PendingFunctionCall = existing ?? { + responseId, + itemId: optionalString(message['item_id']), + callId, + arguments: '', + dispatched: false, + outputSubmitted: false, + responseCompleted: false, + }; + if ( + call.arguments.length + delta.length > + QWEN_REALTIME_LIMITS.maxFunctionArgumentsChars + ) { + protocolError( + 'Realtime function arguments exceeded the allowed size.', + 'function_arguments_too_large', + ); + break; + } + call.arguments += delta; + pendingCalls.set(callId, call); + callback(() => + callbacks.onFunctionArgumentsDelta?.({ + ...eventContext(message), + responseId, + itemId: call.itemId, + callId, + delta, + }), + ); + break; + } + case 'response.function_call_arguments.done': { + const responseId = readResponseId(message); + if (!responseId || !isCurrentResponse(message, type, responseId)) { + break; + } + const callId = optionalString(message['call_id']); + const name = optionalString(message['name']); + const args = optionalString( + message['arguments'], + QWEN_REALTIME_LIMITS.maxFunctionArgumentsChars, + ); + if (!callId || !name || args === undefined) { + protocolError( + 'Realtime function completion was invalid.', + 'invalid_function_arguments', + ); + break; + } + const existing = pendingCalls.get(callId); + if (existing && existing.responseId !== responseId) { + ignoreEvent(message, type, 'stale_call'); + break; + } + if (existing?.dispatched) { + if (existing.name !== name || existing.arguments !== args) { + protocolError( + 'Realtime model changed a handoff call after dispatch.', + 'ambiguous_handoff', + ); + } + break; + } + if ( + !existing && + pendingCalls.size >= QWEN_REALTIME_LIMITS.maxPendingFunctionCalls + ) { + protocolError( + 'Realtime provider created too many pending function calls.', + 'too_many_function_calls', + ); + break; + } + const call: PendingFunctionCall = existing ?? { + responseId, + itemId: optionalString(message['item_id']), + callId, + arguments: '', + dispatched: false, + outputSubmitted: false, + responseCompleted: false, + }; + call.name = name; + pendingCalls.set(callId, call); + dispatchFunctionCall(message, call, args); + break; + } + case 'response.output_item.done': { + const responseId = readResponseId(message); + if (!responseId || !isCurrentResponse(message, type, responseId)) { + break; + } + const item = isRecord(message['item']) ? message['item'] : undefined; + if (item?.['type'] !== 'function_call') break; + const callId = optionalString(item['call_id']); + const name = optionalString(item['name']); + const args = optionalString( + item['arguments'], + QWEN_REALTIME_LIMITS.maxFunctionArgumentsChars, + ); + if (!callId || !name || args === undefined) break; + const existing = pendingCalls.get(callId); + if (existing && existing.responseId !== responseId) { + ignoreEvent(message, type, 'stale_call'); + break; + } + if (existing?.dispatched) { + if (existing.name !== name || existing.arguments !== args) { + protocolError( + 'Realtime model changed a handoff call after dispatch.', + 'ambiguous_handoff', + ); + } + break; + } + if ( + !existing && + pendingCalls.size >= QWEN_REALTIME_LIMITS.maxPendingFunctionCalls + ) { + protocolError( + 'Realtime provider created too many pending function calls.', + 'too_many_function_calls', + ); + break; + } + const call: PendingFunctionCall = existing ?? { + responseId, + itemId: optionalString(item['id']), + callId, + arguments: '', + dispatched: false, + outputSubmitted: false, + responseCompleted: false, + }; + call.name = name; + pendingCalls.set(callId, call); + dispatchFunctionCall(message, call, args); + break; + } + case 'response.done': { + const responseId = + readResponseId(message, true) ?? + readResponseId(message) ?? + activeResponseId; + if (!responseId) { + if (lastCompletedResponseId) { + ignoreEvent(message, type, 'stale_response'); + break; + } + protocolError( + 'Realtime response completion omitted its identifier.', + 'invalid_response', + ); + break; + } + if (cancelledResponseIds.has(responseId)) { + const responseInputItemId = responseInputItemIds.get(responseId); + collectDirectTranscript(responseId); + cancelledResponseIds.delete(responseId); + completePendingCallsForResponse(responseId, 'cancelled'); + consumeResponseInput(responseId); + if (activeResponseId === responseId) { + activeResponseId = undefined; + activeResponseAuthority = undefined; + } + if (activeAudioResponseId === responseId) { + activeAudioResponseId = undefined; + } + lastCompletedResponseId = responseId; + callback(() => + callbacks.onResponseDone?.({ + ...eventContext(message), + responseId, + ...(responseInputItemId + ? { inputItemId: responseInputItemId } + : {}), + status: 'cancelled', + }), + ); + queueMicrotask(flushResponseCreate); + responseAuthorities.delete(responseId); + delegatedResponseIds.delete(responseId); + responseOutputText.delete(responseId); + collectedDirectResponseIds.delete(responseId); + break; + } + if (!isCurrentResponse(message, type, responseId)) break; + const response = isRecord(message['response']) + ? message['response'] + : undefined; + const status = optionalString(response?.['status']); + const responseInputItemId = responseInputItemIds.get(responseId); + if (activeResponseAuthority === 'direct' && status === 'failed') { + const inputLossError = pendingInputLossError(); + if (inputLossError) fail(inputLossError); + if (terminal) break; + } + completePendingCallsForResponse(responseId, status); + collectDirectTranscript(responseId); + lastCompletedResponseId = responseId; + activeResponseId = undefined; + activeResponseAuthority = undefined; + activeAudioResponseId = undefined; + consumeResponseInput(responseId); + responseAuthorities.delete(responseId); + delegatedResponseIds.delete(responseId); + responseOutputText.delete(responseId); + collectedDirectResponseIds.delete(responseId); + callback(() => + callbacks.onResponseDone?.({ + ...eventContext(message), + responseId, + ...(responseInputItemId + ? { inputItemId: responseInputItemId } + : {}), + status, + }), + ); + if (status === 'failed') { + notifyError( + new QwenRealtimeError( + 'Realtime response failed.', + 'response_failed', + false, + ), + ); + } + queueMicrotask(flushResponseCreate); + break; + } + case 'rate_limits.updated': + case 'rate_limit.updated': { + break; + } + case 'error': { + const providerError = isRecord(message['error']) + ? message['error'] + : undefined; + const code = optionalString(providerError?.['code']); + const providerType = optionalString(providerError?.['type']); + const param = optionalString(providerError?.['param']); + const status = optionalHttpStatus( + providerError?.['status'] ?? message['status'], + ); + const errorMessage = sanitizeErrorText( + providerError?.['message'] ?? + providerError?.['code'] ?? + 'Qwen Realtime request failed.', + config.apiKey, + ); + const kind = classifyRealtimeErrorKind(code, errorMessage, status); + fail( + new QwenRealtimeError(errorMessage, code, true, { + kind, + status, + providerType, + param, + }), + ); + break; + } + default: + break; + } + }); + + ws.on('unexpected-response', (...args: unknown[]) => { + const response = isRecord(args[1]) ? args[1] : undefined; + const status = optionalHttpStatus(response?.['statusCode']); + const code = status ? `http_${status}` : 'connection_failed'; + const errorMessage = status + ? `Realtime provider rejected the WebSocket upgrade (${status}).` + : 'Realtime provider rejected the WebSocket upgrade.'; + const kind = classifyRealtimeErrorKind(code, errorMessage, status); + fail( + new QwenRealtimeError(errorMessage, code, true, { + kind, + status, + }), + ); + }); + + ws.on('error', (rawError: unknown) => { + const errorText = sanitizeErrorText( + rawError instanceof Error ? rawError.message : rawError, + config.apiKey, + ); + const statusMatch = /unexpected server response:\s*(\d{3})/i.exec( + errorText, + ); + const status = optionalHttpStatus(statusMatch?.[1]); + fail( + new QwenRealtimeError( + errorText, + status ? `http_${status}` : 'socket_error', + true, + { status }, + ), + ); + }); + + ws.on('close', (...args: unknown[]) => { + clearConnectTimer(); + removeAbortListener(); + if (closedByClient || terminal) return; + const inputLossError = pendingInputLossError(); + if (inputLossError) { + fail(inputLossError); + return; + } + const code = optionalFiniteNumber(args[0]); + const reason = sanitizeErrorText(args[1], config.apiKey); + const suffix = code ? ` (${code}${reason ? `: ${reason}` : ''})` : ''; + const reasonKind = classifyRealtimeErrorKind(undefined, reason); + const error = new QwenRealtimeError( + `Realtime connection closed unexpectedly${suffix}.`, + 'connection_closed', + true, + { + kind: + code !== undefined && [1001, 1006, 1011, 1012, 1013].includes(code) + ? 'transient' + : reasonKind, + closeCode: code, + }, + ); + if (!settled) { + settled = true; + reject(error); + } else { + notifyError(error); + } + terminal = true; + settleClosed({ reason: 'remote', error }); + }); + + abortListener = () => { + fail(new QwenRealtimeError('Realtime connection was aborted.')); + }; + deps.abortSignal?.addEventListener('abort', abortListener, { once: true }); + if (deps.abortSignal?.aborted) abortListener(); + + connectTimer = setTimeout(() => { + if (!ready) { + fail( + new QwenRealtimeError( + 'Realtime connection timed out.', + 'connection_timeout', + ), + ); + } + }, connectTimeoutMs); + }); +} diff --git a/packages/qwen-live/src/realtime/sanitize.test.ts b/packages/qwen-live/src/realtime/sanitize.test.ts new file mode 100644 index 00000000000..c43783b2236 --- /dev/null +++ b/packages/qwen-live/src/realtime/sanitize.test.ts @@ -0,0 +1,29 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it } from 'vitest'; +import { escapeAnsiCtrlCodes } from './sanitize.js'; + +describe('escapeAnsiCtrlCodes', () => { + it('escapes ANSI escape sequences into their JSON-escaped form', () => { + expect(escapeAnsiCtrlCodes('\u001b[31mred\u001b[0m')).toBe( + '\\u001b[31mred\\u001b[0m', + ); + }); + + it('escapes consistently on repeated calls despite the shared global regex', () => { + // The module holds one global regex whose lastIndex must be reset + // between uses; a stale lastIndex would skip leading matches. + expect(escapeAnsiCtrlCodes('\u001b[2Jcleared')).toBe('\\u001b[2Jcleared'); + expect(escapeAnsiCtrlCodes('\u001b[2Jcleared')).toBe('\\u001b[2Jcleared'); + }); + + it('returns clean strings unchanged', () => { + const clean = 'Realtime provider rejected the request (429).'; + expect(escapeAnsiCtrlCodes(clean)).toBe(clean); + expect(escapeAnsiCtrlCodes('')).toBe(''); + }); +}); diff --git a/packages/qwen-live/src/realtime/sanitize.ts b/packages/qwen-live/src/realtime/sanitize.ts new file mode 100644 index 00000000000..526de991f7e --- /dev/null +++ b/packages/qwen-live/src/realtime/sanitize.ts @@ -0,0 +1,23 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import ansiRegex from 'ansi-regex'; + +const regex = ansiRegex(); + +/** + * Ported from packages/cli/src/ui/utils/textUtils.ts (escapeAnsiCtrlCodes), + * reduced to the string path this package actually uses: escape ANSI/control + * sequences in untrusted text before it reaches logs or model context. + * Returns the input unchanged when it contains no ANSI sequences. + */ +export function escapeAnsiCtrlCodes(text: string): string { + if (text.search(regex) === -1) { + return text; // No changes return original string + } + regex.lastIndex = 0; // needed for global regex + return text.replace(regex, (match) => JSON.stringify(match).slice(1, -1)); +} diff --git a/packages/qwen-live/src/realtime/socket.ts b/packages/qwen-live/src/realtime/socket.ts new file mode 100644 index 00000000000..e259b1c29c1 --- /dev/null +++ b/packages/qwen-live/src/realtime/socket.ts @@ -0,0 +1,37 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Minimal socket abstraction shared by the realtime client and its tests. + * + * Ported verbatim from packages/cli/src/ui/voice/voice-stream-session.ts so + * the ported realtime client (and its FakeSocket-based tests) keep the exact + * injection seam they were written against. + */ +export interface SocketLike { + readyState: number; + OPEN: number; + bufferedAmount?: number; + send: (data: string | Uint8Array) => void; + close: () => void; + on: (event: string, cb: (...args: unknown[]) => void) => void; +} + +/** + * Derive the WebSocket origin for a DashScope-style HTTP base URL: swap the + * scheme and strip a trailing `/compatible-mode/v1` or `/v1` path prefix. + */ +export function deriveWebSocketBase(baseUrl: string): string { + const url = new URL(baseUrl); + const wsScheme = url.protocol === 'https:' ? 'wss:' : 'ws:'; + let prefix = url.pathname.replace(/\/+$/, ''); + if (prefix.endsWith('/compatible-mode/v1')) { + prefix = prefix.slice(0, -'/compatible-mode/v1'.length); + } else if (prefix.endsWith('/v1')) { + prefix = prefix.slice(0, -'/v1'.length); + } + return `${wsScheme}//${url.host}${prefix}`; +} diff --git a/packages/qwen-live/src/tools/definitions.ts b/packages/qwen-live/src/tools/definitions.ts new file mode 100644 index 00000000000..bdaa9e56559 --- /dev/null +++ b/packages/qwen-live/src/tools/definitions.ts @@ -0,0 +1,213 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * The realtime model's tool surface: seven receipt-style dispatch tools plus + * remain_silent. Descriptions encode the two disciplines every tool obeys: + * tools return receipts and snapshots (never long-task results — those flow + * back through injection), and the model must not claim work happened + * without a receipt. + */ + +import { + REMAIN_SILENT_TOOL_NAME, + type RealtimeToolDefinition, +} from '../realtime/realtime-session.js'; + +export const APPSHOT_TOOL_NAME = 'appshot'; +export const SESSION_LIST_TOOL_NAME = 'session_list'; +export const SESSION_CREATE_TOOL_NAME = 'session_create'; +export const HANDOFF_TOOL_NAME = 'handoff'; +export const SESSION_MONITOR_TOOL_NAME = 'session_monitor'; +export const SESSION_STOP_TOOL_NAME = 'session_stop'; +export const RESPOND_PERMISSION_TOOL_NAME = 'respond_permission'; + +const APPSHOT_TOOL: RealtimeToolDefinition = { + type: 'function', + function: { + name: APPSHOT_TOOL_NAME, + description: + 'Capture what the user currently sees on screen. Returns the frontmost ' + + 'app, window title, and an accessibility-text summary, and registers ' + + 'the screenshot as an asset you can attach to a later handoff via ' + + 'input_refs. Use it whenever the user refers to "this", "the screen", ' + + 'or visible content.', + parameters: { type: 'object', properties: {}, additionalProperties: false }, + }, +}; + +const SESSION_LIST_TOOL: RealtimeToolDefinition = { + type: 'function', + function: { + name: SESSION_LIST_TOOL_NAME, + description: + 'List the coding sessions you can dispatch work to, with their short ' + + 'handles, working directories, and whether each is idle or busy. Call ' + + 'this before referring to any session you have not listed yet in this ' + + 'call.', + parameters: { type: 'object', properties: {}, additionalProperties: false }, + }, +}; + +const SESSION_CREATE_TOOL: RealtimeToolDefinition = { + type: 'function', + function: { + name: SESSION_CREATE_TOOL_NAME, + description: + 'Create a new coding session. Only needed when the user explicitly ' + + 'wants separate parallel workstreams; handoff without a session picks ' + + 'or creates a sensible default on its own.', + parameters: { + type: 'object', + properties: { + label: { + type: 'string', + description: 'Optional human label for the session.', + }, + cwd: { + type: 'string', + description: 'Optional working directory for the session.', + }, + }, + additionalProperties: false, + }, + }, +}; + +const HANDOFF_TOOL: RealtimeToolDefinition = { + type: 'function', + function: { + name: HANDOFF_TOOL_NAME, + description: + "Send the user's request to a coding session for execution. This is " + + 'the default action for anything that touches files, runs commands, ' + + 'needs the screen inspected in depth, or requires up-to-date ' + + "information. Pass the user's own words in `task`; do not rewrite " + + 'them. Returns a receipt immediately — the result arrives later as a ' + + '[COMPLETE] context message. Targeting a busy session appends the ' + + 'instruction to its running task (the receipt says how it landed). ' + + 'Before your first tool call in a user turn, say one short neutral ' + + 'sentence about what you are doing; never promise the outcome.', + parameters: { + type: 'object', + properties: { + task: { + type: 'string', + description: "The user's request, in their own words.", + }, + session: { + type: 'string', + description: + 'Target session handle from session_list (e.g. "session_1"). ' + + 'Omit to use the default session.', + }, + input_refs: { + type: 'array', + items: { type: 'string' }, + description: + 'Asset ids to attach (e.g. an appshot screenshot: "asset_1").', + }, + }, + required: ['task'], + additionalProperties: false, + }, + }, + capturesTranscript: true, +}; + +const SESSION_MONITOR_TOOL: RealtimeToolDefinition = { + type: 'function', + function: { + name: SESSION_MONITOR_TOOL_NAME, + description: + 'Get a progress snapshot for a session or job (state plus recent ' + + 'activity). Use it only when the user asks how something is going; ' + + 'completed work announces itself without polling.', + parameters: { + type: 'object', + properties: { + session: { type: 'string', description: 'Session handle.' }, + job: { type: 'string', description: 'Job reference (e.g. "job_2").' }, + }, + additionalProperties: false, + }, + }, +}; + +const SESSION_STOP_TOOL: RealtimeToolDefinition = { + type: 'function', + function: { + name: SESSION_STOP_TOOL_NAME, + description: + 'Cancel the running task in a session. This is the only way to stop ' + + 'work: the user interrupting your speech never cancels tasks. Call it ' + + 'only when the user clearly asks to stop or abandon the work.', + parameters: { + type: 'object', + properties: { + session: { type: 'string', description: 'Session handle.' }, + job: { type: 'string', description: 'Job reference.' }, + }, + additionalProperties: false, + }, + }, +}; + +const RESPOND_PERMISSION_TOOL: RealtimeToolDefinition = { + type: 'function', + function: { + name: RESPOND_PERMISSION_TOOL_NAME, + description: + "Relay the user's spoken answer to a pending [PERMISSION] request. " + + '`allow_always` also lets similar requests through silently for a ' + + 'while. Only call this after the user actually answered; never decide ' + + 'for them.', + parameters: { + type: 'object', + properties: { + request_id: { + type: 'string', + description: 'The id from the [PERMISSION] message (e.g. "req_3").', + }, + decision: { + type: 'string', + enum: ['allow', 'allow_always', 'deny'], + }, + note: { + type: 'string', + description: + "Optional constraint the user added ('only this file'); it is " + + 'relayed to the coding session together with the vote.', + }, + }, + required: ['request_id', 'decision'], + additionalProperties: false, + }, + }, +}; + +const REMAIN_SILENT_TOOL: RealtimeToolDefinition = { + type: 'function', + function: { + name: REMAIN_SILENT_TOOL_NAME, + description: + 'Call this when the best response is to say nothing. Use it instead ' + + 'of speaking after silent context messages whenever acknowledging ' + + 'aloud would be distracting. This tool has no user-visible effect.', + parameters: { type: 'object', properties: {}, additionalProperties: false }, + }, +}; + +export const LIVE_SESSION_TOOLS: readonly RealtimeToolDefinition[] = [ + APPSHOT_TOOL, + SESSION_LIST_TOOL, + SESSION_CREATE_TOOL, + HANDOFF_TOOL, + SESSION_MONITOR_TOOL, + SESSION_STOP_TOOL, + RESPOND_PERMISSION_TOOL, + REMAIN_SILENT_TOOL, +]; diff --git a/packages/qwen-live/src/tools/dispatcher.test.ts b/packages/qwen-live/src/tools/dispatcher.test.ts new file mode 100644 index 00000000000..d3d80f733ea --- /dev/null +++ b/packages/qwen-live/src/tools/dispatcher.test.ts @@ -0,0 +1,206 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { ToolDispatcher } from './dispatcher.js'; +import type { ToolContext, ToolHandler } from './dispatcher.js'; + +function makeContext( + transcript: ToolContext['activeTranscript'] = [], +): ToolContext { + return { activeTranscript: transcript }; +} + +function makeDispatcher( + handlers: Record, + timeoutMs?: number, +): ToolDispatcher { + return new ToolDispatcher({ + handlers: new Map(Object.entries(handlers)), + ...(timeoutMs !== undefined ? { timeoutMs } : {}), + }); +} + +afterEach(() => { + vi.useRealTimers(); +}); + +describe('ToolDispatcher', () => { + it('answers an unregistered tool with an error receipt instead of stalling', async () => { + const dispatcher = makeDispatcher({}); + + const result = await dispatcher.dispatch('mystery', '{}', makeContext()); + + expect(result.ok).toBe(false); + const receipt = JSON.parse(result.receipt) as { + status: string; + note: string; + }; + expect(receipt.status).toBe('error'); + expect(receipt.note).toContain('No handler'); + expect(receipt.note).toContain('mystery'); + }); + + it.each([ + ['invalid JSON', '{not json'], + ['empty string', ''], + ['whitespace only', ' '], + ['a JSON array', '[1, 2, 3]'], + ['a JSON scalar', '"hello"'], + ])('degrades %s arguments to an empty object', async (_label, raw) => { + const seen: Array> = []; + const dispatcher = makeDispatcher({ + echo: (args) => { + seen.push(args); + return { status: 'ok' }; + }, + }); + + const result = await dispatcher.dispatch('echo', raw, makeContext()); + + expect(result.ok).toBe(true); + expect(seen).toEqual([{}]); + }); + + it('passes well-formed JSON object arguments through to the handler', async () => { + const seen: Array> = []; + const dispatcher = makeDispatcher({ + echo: (args) => { + seen.push(args); + return { status: 'ok' }; + }, + }); + + await dispatcher.dispatch( + 'echo', + '{"sessionHandle":"session_1","count":2}', + makeContext(), + ); + + expect(seen).toEqual([{ sessionHandle: 'session_1', count: 2 }]); + }); + + it('serializes a successful handler result as the JSON receipt', async () => { + const outcome = { status: 'accepted', jobHandle: 'job_1', queued: false }; + const dispatcher = makeDispatcher({ + start_job: () => outcome, + }); + + const result = await dispatcher.dispatch('start_job', '{}', makeContext()); + + expect(result.ok).toBe(true); + expect(result.receipt).toBe(JSON.stringify(outcome)); + }); + + it('awaits async handlers before building the receipt', async () => { + const dispatcher = makeDispatcher({ + async_tool: async () => ({ status: 'done' }), + }); + + const result = await dispatcher.dispatch('async_tool', '{}', makeContext()); + + expect(result.ok).toBe(true); + expect(JSON.parse(result.receipt)).toEqual({ status: 'done' }); + }); + + it('turns a thrown handler error into an error receipt with the message truncated to 300 chars', async () => { + const longMessage = 'x'.repeat(400); + const dispatcher = makeDispatcher({ + boom: () => { + throw new Error(longMessage); + }, + }); + + const result = await dispatcher.dispatch('boom', '{}', makeContext()); + + expect(result.ok).toBe(false); + const receipt = JSON.parse(result.receipt) as { + status: string; + note: string; + }; + expect(receipt.status).toBe('error'); + expect(receipt.note).toBe(longMessage.slice(0, 300)); + expect(receipt.note).toHaveLength(300); + }); + + it('falls back to a generic note for non-Error and empty-message throws', async () => { + const dispatcher = makeDispatcher({ + raw_throw: () => { + // A non-Error throw (object, not string — the lint bans literals). + throw { reason: 'not an error object' }; + }, + empty_message: () => { + throw new Error(''); + }, + }); + + for (const name of ['raw_throw', 'empty_message']) { + const result = await dispatcher.dispatch(name, '{}', makeContext()); + expect(result.ok).toBe(false); + expect(JSON.parse(result.receipt)).toEqual({ + status: 'error', + note: 'the tool failed', + }); + } + }); + + it('times out a hung handler and returns the timeout receipt', async () => { + vi.useFakeTimers(); + const dispatcher = makeDispatcher( + { + hang: () => new Promise>(() => {}), + }, + 50, + ); + + const pending = dispatcher.dispatch('hang', '{}', makeContext()); + await vi.advanceTimersByTimeAsync(50); + const result = await pending; + + expect(result.ok).toBe(false); + const receipt = JSON.parse(result.receipt) as { + status: string; + note: string; + }; + // The handler keeps running (side effects are real): the receipt must + // read as "still in progress", never as a retryable failure. + expect(receipt.status).toBe('pending'); + expect(receipt.note).toContain('Do not retry'); + }); + + it('does not time out a handler that resolves before the deadline', async () => { + vi.useFakeTimers(); + const dispatcher = makeDispatcher( + { + quick: () => ({ status: 'ok' }), + }, + 50, + ); + + const result = await dispatcher.dispatch('quick', '{}', makeContext()); + + expect(result.ok).toBe(true); + expect(JSON.parse(result.receipt)).toEqual({ status: 'ok' }); + }); + + it('threads ctx.activeTranscript through to the handler untouched', async () => { + const transcript = [ + { role: 'user' as const, text: 'run the tests' }, + { role: 'assistant' as const, text: 'Starting them now.' }, + ]; + let received: ToolContext | undefined; + const dispatcher = makeDispatcher({ + capture: (_args, ctx) => { + received = ctx; + return { status: 'ok' }; + }, + }); + + await dispatcher.dispatch('capture', '{}', makeContext(transcript)); + + expect(received?.activeTranscript).toBe(transcript); + }); +}); diff --git a/packages/qwen-live/src/tools/dispatcher.ts b/packages/qwen-live/src/tools/dispatcher.ts new file mode 100644 index 00000000000..a52d89853c9 --- /dev/null +++ b/packages/qwen-live/src/tools/dispatcher.ts @@ -0,0 +1,124 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Routes realtime function calls to handlers and turns whatever happens — + * success, handler error, bad arguments, timeout — into a JSON receipt + * string. No call is ever left unanswered: an unanswered function call + * would stall the realtime response arbitration. + */ + +/** + * Generous by design: the timeout is a last-resort answer for the model, + * not a cancellation — the handler keeps running and its side effects + * (a created job, a submitted prompt) stay real. The receipt wording must + * therefore steer the model away from retrying. + */ +const DEFAULT_HANDLER_TIMEOUT_MS = 30_000; +const TIMEOUT_NOTE = + 'The action is still running in the background and may yet finish. Do ' + + 'not retry the call; check on it with session_monitor.'; + +/** Per-call context threaded through from the realtime function-call event. */ +export interface ToolContext { + /** Transcript tail captured at call time (capturesTranscript tools). */ + activeTranscript: ReadonlyArray<{ role: 'user' | 'assistant'; text: string }>; +} + +export type ToolHandler = ( + args: Record, + ctx: ToolContext, +) => Promise> | Record; + +export interface ToolDispatchResult { + /** JSON receipt to submit as the function call output. */ + receipt: string; + ok: boolean; +} + +export interface ToolDispatcherOptions { + handlers: ReadonlyMap; + timeoutMs?: number; +} + +function parseArguments(raw: string): Record { + if (!raw.trim()) return {}; + try { + const parsed = JSON.parse(raw) as unknown; + if (typeof parsed === 'object' && parsed !== null && !Array.isArray(parsed)) + return parsed as Record; + } catch { + /* fall through to the lenient default */ + } + return {}; +} + +class ToolTimeoutError extends Error { + constructor() { + super('tool handler timed out'); + this.name = 'ToolTimeoutError'; + } +} + +export class ToolDispatcher { + private readonly handlers: ReadonlyMap; + private readonly timeoutMs: number; + + constructor(options: ToolDispatcherOptions) { + this.handlers = options.handlers; + this.timeoutMs = options.timeoutMs ?? DEFAULT_HANDLER_TIMEOUT_MS; + } + + async dispatch( + name: string, + rawArguments: string, + ctx: ToolContext, + ): Promise { + const handler = this.handlers.get(name); + if (!handler) { + return { + ok: false, + receipt: JSON.stringify({ + status: 'error', + note: `No handler for tool ${name}.`, + }), + }; + } + const args = parseArguments(rawArguments); + let timer: ReturnType | undefined; + try { + const outcome = await Promise.race([ + Promise.resolve(handler(args, ctx)), + new Promise((_, reject) => { + timer = setTimeout(() => { + reject(new ToolTimeoutError()); + }, this.timeoutMs); + timer.unref?.(); + }), + ]); + return { ok: true, receipt: JSON.stringify(outcome) }; + } catch (error) { + if (error instanceof ToolTimeoutError) { + return { + ok: false, + receipt: JSON.stringify({ status: 'pending', note: TIMEOUT_NOTE }), + }; + } + return { + ok: false, + receipt: JSON.stringify({ + status: 'error', + note: + error instanceof Error && error.message + ? error.message.slice(0, 300) + : 'the tool failed', + }), + }; + } finally { + if (timer !== undefined) clearTimeout(timer); + } + } +} diff --git a/packages/qwen-live/src/tools/handles.test.ts b/packages/qwen-live/src/tools/handles.test.ts new file mode 100644 index 00000000000..7048008109f --- /dev/null +++ b/packages/qwen-live/src/tools/handles.test.ts @@ -0,0 +1,210 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { HandleRegistry } from './handles.js'; +import type { JobRecord } from './handles.js'; +import type { BackendHandle } from '../adaptor/types.js'; + +function backend(id: string, adaptor = 'qwen-code'): BackendHandle { + return { id, adaptor }; +} + +beforeEach(() => { + vi.useFakeTimers(); + vi.setSystemTime(1_000_000); +}); + +afterEach(() => { + vi.useRealTimers(); +}); + +describe('HandleRegistry sessions', () => { + it('returns the same handle for the same backend session (idempotent)', () => { + const registry = new HandleRegistry(); + + const first = registry.session(backend('abc')); + const again = registry.session(backend('abc')); + + expect(first).toBe('session_1'); + expect(again).toBe(first); + }); + + it('assigns incrementing handles to distinct backends', () => { + const registry = new HandleRegistry(); + + expect(registry.session(backend('abc'))).toBe('session_1'); + expect(registry.session(backend('def'))).toBe('session_2'); + // Same id under a different adaptor is a different backend. + expect(registry.session(backend('abc', 'acp'))).toBe('session_3'); + }); + + it('resolves session handles with surrounding whitespace trimmed', () => { + const registry = new HandleRegistry(); + const handle = registry.session(backend('abc')); + + expect(registry.resolveSession(` ${handle} `)).toEqual(backend('abc')); + expect(registry.resolveSession('session_99')).toBeUndefined(); + }); +}); + +describe('HandleRegistry jobs', () => { + it('creates jobs with incrementing handles and accepted state', () => { + const registry = new HandleRegistry(); + const sessionHandle = registry.session(backend('abc')); + + const job = registry.createJob({ + sessionHandle, + backend: backend('abc'), + jobRef: 'prompt-1', + task: 'run the tests', + }); + + expect(job).toEqual({ + jobHandle: 'job_1', + sessionHandle, + backend: backend('abc'), + jobRef: 'prompt-1', + state: 'accepted', + task: 'run the tests', + createdAt: 1_000_000, + }); + expect( + registry.createJob({ + sessionHandle, + backend: backend('abc'), + task: 'another task', + }).jobHandle, + ).toBe('job_2'); + }); + + it('omits jobRef when none was supplied', () => { + const registry = new HandleRegistry(); + const job = registry.createJob({ + sessionHandle: 'session_1', + backend: backend('abc'), + task: 'no ref', + }); + + expect('jobRef' in job).toBe(false); + expect(registry.jobByRef('prompt-x')).toBeUndefined(); + }); + + it('returns the existing record instead of minting a duplicate for a known jobRef', () => { + const registry = new HandleRegistry(); + const first = registry.createJob({ + sessionHandle: 'session_1', + backend: backend('abc'), + jobRef: 'prompt-1', + task: 'original task', + }); + first.state = 'running'; + + // A steer that joined the running turn reports the SAME jobRef: the + // registry must not orphan the running job behind a second record. + const again = registry.createJob({ + sessionHandle: 'session_1', + backend: backend('abc'), + jobRef: 'prompt-1', + task: 'joined instruction', + }); + + expect(again).toBe(first); + expect(again.jobHandle).toBe('job_1'); + expect(registry.resolveJob('job_2')).toBeUndefined(); + expect(registry.activeJobForSession('session_1')).toBe(first); + + first.state = 'done'; + expect(registry.activeJobForSession('session_1')).toBeUndefined(); + }); + + it('resolves jobs by trimmed handle and by backend jobRef', () => { + const registry = new HandleRegistry(); + const job = registry.createJob({ + sessionHandle: 'session_1', + backend: backend('abc'), + jobRef: 'prompt-7', + task: 'build', + }); + + expect(registry.resolveJob(` ${job.jobHandle} `)).toBe(job); + expect(registry.resolveJob('job_99')).toBeUndefined(); + expect(registry.jobByRef('prompt-7')).toBe(job); + expect(registry.jobByRef('prompt-8')).toBeUndefined(); + }); + + it('picks the newest non-terminal job as the active job for a session', () => { + const registry = new HandleRegistry(); + const create = (task: string): JobRecord => { + vi.advanceTimersByTime(1_000); + return registry.createJob({ + sessionHandle: 'session_1', + backend: backend('abc'), + task, + }); + }; + + const older = create('older'); + const newer = create('newer'); + expect(registry.activeJobForSession('session_1')?.task).toBe('newer'); + + newer.state = 'done'; + expect(registry.activeJobForSession('session_1')?.task).toBe('older'); + + older.state = 'failed'; + expect(registry.activeJobForSession('session_1')).toBeUndefined(); + + const cancelled = create('cancelled one'); + cancelled.state = 'cancelled'; + expect(registry.activeJobForSession('session_1')).toBeUndefined(); + + const running = create('running one'); + running.state = 'running'; + expect(registry.activeJobForSession('session_1')).toBe(running); + }); + + it('scopes the active job lookup to the requested session', () => { + const registry = new HandleRegistry(); + registry.createJob({ + sessionHandle: 'session_1', + backend: backend('abc'), + task: 'for session 1', + }); + + expect(registry.activeJobForSession('session_2')).toBeUndefined(); + }); +}); + +describe('HandleRegistry assets', () => { + it('registers assets and resolves trimmed handles', () => { + const registry = new HandleRegistry(); + const asset = registry.registerAsset({ + path: '/tmp/shot.png', + mimeType: 'image/png', + }); + + expect(asset.assetHandle).toBe('asset_1'); + expect(registry.resolveAsset(' asset_1 ')).toBe(asset); + }); + + it('keeps at most 32 assets, evicting the oldest first', () => { + const registry = new HandleRegistry(); + for (let index = 1; index <= 33; index++) { + registry.registerAsset({ + path: `/tmp/shot-${index}.png`, + mimeType: 'image/png', + }); + } + + expect(registry.resolveAsset('asset_1')).toBeUndefined(); + expect(registry.resolveAsset('asset_2')?.path).toBe('/tmp/shot-2.png'); + expect(registry.resolveAsset('asset_33')?.path).toBe('/tmp/shot-33.png'); + + registry.registerAsset({ path: '/tmp/shot-34.png', mimeType: 'image/png' }); + expect(registry.resolveAsset('asset_2')).toBeUndefined(); + expect(registry.resolveAsset('asset_3')).toBeDefined(); + }); +}); diff --git a/packages/qwen-live/src/tools/handles.ts b/packages/qwen-live/src/tools/handles.ts new file mode 100644 index 00000000000..1badd9989ff --- /dev/null +++ b/packages/qwen-live/src/tools/handles.ts @@ -0,0 +1,161 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Short-handle registries. The realtime model refers to sessions as + * `session_N`, jobs as `job_N`, and screen-capture assets as `asset_N` — + * short, speakable, and hard to mistranscribe. These maps own the + * bidirectional translation to the real identities. + */ + +import type { BackendHandle } from '../adaptor/types.js'; + +export interface JobRecord { + jobHandle: string; + sessionHandle: string; + backend: BackendHandle; + /** Adaptor-side correlation id (qwen serve: promptId). */ + jobRef?: string; + state: 'accepted' | 'running' | 'done' | 'failed' | 'cancelled'; + task: string; + createdAt: number; +} + +export interface AssetRecord { + assetHandle: string; + path: string; + mimeType: string; + createdAt: number; +} + +const MAX_ASSETS = 32; + +export class HandleRegistry { + private readonly sessions = new Map(); + private readonly sessionsByBackendId = new Map(); + /** Sessions whose backend reported closure — handles no longer resolve. */ + private readonly closedSessions = new Set(); + private readonly jobs = new Map(); + private readonly jobsByRef = new Map(); + private readonly assets = new Map(); + private sessionSeq = 0; + private jobSeq = 0; + private assetSeq = 0; + + /** Register (or return the existing handle for) a backend session. */ + session(backend: BackendHandle): string { + const key = `${backend.adaptor}:${backend.id}`; + const existing = this.sessionsByBackendId.get(key); + if (existing) return existing; + const handle = `session_${++this.sessionSeq}`; + this.sessions.set(handle, backend); + this.sessionsByBackendId.set(key, handle); + return handle; + } + + resolveSession(handle: string): BackendHandle | undefined { + const key = handle.trim(); + if (this.closedSessions.has(key)) return undefined; + return this.sessions.get(key); + } + + /** Mark a session closed: its handle stops resolving from now on. */ + closeSession(handle: string): void { + this.closedSessions.add(handle.trim()); + } + + /** + * Register a new job. Idempotent by jobRef: a jobRef already registered + * means the backend attributed the prompt to that existing turn (e.g. a + * steer that joined it), so the existing record is returned instead of + * minting a second job that would orphan the first in 'running'. + */ + createJob(fields: { + sessionHandle: string; + backend: BackendHandle; + jobRef?: string; + task: string; + }): JobRecord { + if (fields.jobRef !== undefined) { + const existing = this.jobByRef(fields.jobRef); + if (existing) return existing; + } + const record: JobRecord = { + jobHandle: `job_${++this.jobSeq}`, + sessionHandle: fields.sessionHandle, + backend: fields.backend, + ...(fields.jobRef !== undefined ? { jobRef: fields.jobRef } : {}), + state: 'accepted', + task: fields.task, + createdAt: Date.now(), + }; + this.jobs.set(record.jobHandle, record); + if (record.jobRef !== undefined) { + this.jobsByRef.set(record.jobRef, record.jobHandle); + } + return record; + } + + resolveJob(handle: string): JobRecord | undefined { + return this.jobs.get(handle.trim()); + } + + /** Find the job a backend event's jobRef belongs to. */ + jobByRef(jobRef: string): JobRecord | undefined { + const handle = this.jobsByRef.get(jobRef); + return handle ? this.jobs.get(handle) : undefined; + } + + /** + * Reconcile a session's non-terminal jobs against reality: the backend + * reports idle, so any 'accepted'/'running' record left over from a + * missed terminal event (emitted while no pump was subscribed — pumps + * are per-call and aborted at call end) transitions to 'done'. Gated on + * the caller's isBusy check so a genuinely busy backend is never + * touched. + */ + reconcileIdleSession(sessionHandle: string): void { + for (const job of this.jobs.values()) { + if (job.sessionHandle !== sessionHandle) continue; + if (job.state === 'done' || job.state === 'failed') continue; + if (job.state === 'cancelled') continue; + job.state = 'done'; + } + } + + /** The most recent non-terminal job for a session, if any. */ + activeJobForSession(sessionHandle: string): JobRecord | undefined { + let candidate: JobRecord | undefined; + for (const job of this.jobs.values()) { + if (job.sessionHandle !== sessionHandle) continue; + if (job.state === 'done' || job.state === 'failed') continue; + if (job.state === 'cancelled') continue; + if (!candidate || job.createdAt > candidate.createdAt) candidate = job; + } + return candidate; + } + + registerAsset(fields: { path: string; mimeType: string }): AssetRecord { + const record: AssetRecord = { + assetHandle: `asset_${++this.assetSeq}`, + path: fields.path, + mimeType: fields.mimeType, + createdAt: Date.now(), + }; + this.assets.set(record.assetHandle, record); + // Bounded: assets reference temp files with short TTLs; keep the newest. + while (this.assets.size > MAX_ASSETS) { + const oldest = this.assets.keys().next().value; + if (oldest === undefined) break; + this.assets.delete(oldest); + } + return record; + } + + resolveAsset(handle: string): AssetRecord | undefined { + return this.assets.get(handle.trim()); + } +} diff --git a/packages/qwen-live/tsconfig.build.json b/packages/qwen-live/tsconfig.build.json new file mode 100644 index 00000000000..82c35366325 --- /dev/null +++ b/packages/qwen-live/tsconfig.build.json @@ -0,0 +1,9 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + "tsBuildInfoFile": "./tsconfig.build.tsbuildinfo", + "sourceMap": false, + "declarationMap": false + }, + "exclude": ["node_modules", "dist", "src/**/*.test.ts", "src/testing/**"] +} diff --git a/packages/qwen-live/tsconfig.json b/packages/qwen-live/tsconfig.json new file mode 100644 index 00000000000..399d1d1e8da --- /dev/null +++ b/packages/qwen-live/tsconfig.json @@ -0,0 +1,11 @@ +{ + "extends": "../../tsconfig.json", + "compilerOptions": { + "outDir": "dist", + "rootDir": "src", + "composite": true, + "types": ["node", "vitest/globals"] + }, + "include": ["src/**/*.ts"], + "exclude": ["node_modules", "dist"] +} diff --git a/packages/qwen-live/vitest.config.ts b/packages/qwen-live/vitest.config.ts new file mode 100644 index 00000000000..c3cdc6f5c63 --- /dev/null +++ b/packages/qwen-live/vitest.config.ts @@ -0,0 +1,29 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { defineConfig } from 'vitest/config'; + +const here = path.dirname(fileURLToPath(import.meta.url)); + +export default defineConfig({ + resolve: { + alias: { + // Read the SDK's TypeScript source directly so tests never depend on a + // stale dist build (same rationale as packages/cli/vitest.config.ts). + '@qwen-code/sdk': path.resolve(here, '../sdk-typescript/src/index.ts'), + }, + }, + test: { + environment: 'node', + include: ['src/**/*.test.ts'], + // Discovery tests exercise real file locking; orchestrator tests use fake + // timers but spawn no processes. Keep generous ceilings for slow CI hosts. + testTimeout: 60_000, + hookTimeout: 60_000, + }, +}); diff --git a/packages/vscode-ide-companion/src/diff-manager.ts b/packages/vscode-ide-companion/src/diff-manager.ts index f1fecfcadfa..b9dc573ddb1 100644 --- a/packages/vscode-ide-companion/src/diff-manager.ts +++ b/packages/vscode-ide-companion/src/diff-manager.ts @@ -103,10 +103,12 @@ export class DiffManager { const rightUri = rightDocUri.toString(); for (const group of vscode.window.tabGroups.all) { const containsDiff = group.tabs.some((tab) => { - const input = tab.input as { - original?: vscode.Uri; - modified?: vscode.Uri; - } | undefined; + const input = tab.input as + | { + original?: vscode.Uri; + modified?: vscode.Uri; + } + | undefined; return ( input?.original?.toString() === leftUri && input?.modified?.toString() === rightUri diff --git a/scripts/build.js b/scripts/build.js index abeae2f8d33..2fa04f0635c 100644 --- a/scripts/build.js +++ b/scripts/build.js @@ -74,6 +74,7 @@ const buildOrder = [ ...(cliOnly ? [] : [ + 'packages/qwen-live', 'packages/webui', 'packages/web-shell', 'packages/vscode-ide-companion', diff --git a/scripts/tests/no-ak-integration-ci.test.js b/scripts/tests/no-ak-integration-ci.test.js index 5c68093c6e3..6c3cfbbe95e 100644 --- a/scripts/tests/no-ak-integration-ci.test.js +++ b/scripts/tests/no-ak-integration-ci.test.js @@ -145,6 +145,10 @@ describe('no-AK integration CI wiring', () => { './fake-openai-server.test.ts', './test-helper.test.ts', './chat-transcript-contract.test.ts', + './qwen-live-m1-call.test.ts', + './qwen-live-m2-inject.test.ts', + './qwen-live-m2-permission.test.ts', + './qwen-live-m2-steering.test.ts', './cli/daemon-invocation-context.test.ts', './cli/list_directory.test.ts', './cli/qwen-serve-routes.test.ts', diff --git a/scripts/tests/package-scripts.test.js b/scripts/tests/package-scripts.test.js index 586c5b7ccec..8868b1777d2 100644 --- a/scripts/tests/package-scripts.test.js +++ b/scripts/tests/package-scripts.test.js @@ -42,10 +42,29 @@ describe('package scripts', () => { " '@qwen-code/sdk',\n" + " '@qwen-code/mobile-mcp',\n" + " '@qwen-code/node-repl-mcp',\n" + + " '@qwen-code/qwen-live',\n" + '];', ); }); + it('builds the standalone qwen-live daemon in the root build order', () => { + const buildScript = readFileSync( + path.join(root, 'scripts/build.js'), + 'utf8', + ); + + // The qwen-live e2e harness spawns packages/qwen-live/dist/index.js and + // the workspace unit tests run from src, so this pin is what catches the + // root build silently dropping the package. + const startIndex = buildScript.indexOf('const buildOrder = ['); + expect(startIndex).toBeGreaterThan(-1); + const buildOrder = buildScript.slice( + startIndex, + buildScript.indexOf('];', startIndex), + ); + expect(buildOrder).toContain("'packages/qwen-live',"); + }); + it('keeps the serve fast-path bundle check outside unit test scripts', () => { const packageJson = readPackageJson(); diff --git a/scripts/version.js b/scripts/version.js index 23f8e569b06..598cf9fad8f 100644 --- a/scripts/version.js +++ b/scripts/version.js @@ -43,11 +43,13 @@ if (!versionType) { run(`npm version ${versionType} --no-git-tag-version --allow-same-version`); // 3. Get all workspaces and filter out the one we don't want to version. -// We intend to maintain sdk, mobile-mcp, and node-repl versions independently. +// We intend to maintain sdk, mobile-mcp, node-repl, and qwen-live versions +// independently. const workspacesToExclude = [ '@qwen-code/sdk', '@qwen-code/mobile-mcp', '@qwen-code/node-repl-mcp', + '@qwen-code/qwen-live', ]; const workspaceNames = JSON.parse( execSync('npm pkg get name --workspaces --json').toString(), diff --git a/vitest.config.ts b/vitest.config.ts index f74b58bbd3d..4cb2185be92 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -8,6 +8,7 @@ export default defineConfig({ 'packages/vscode-ide-companion', 'packages/sdk-typescript', 'packages/node-repl', + 'packages/qwen-live', 'packages/channels/base', 'packages/channels/dingtalk', 'packages/channels/dws',