Files
qwen-code/packages/qwen-live/src/tools/handles.ts
T
8290c81eea feat(qwen-live): standalone voice daemon package — M1 + M2 (#10367)
* feat(qwen-live): scaffold standalone live daemon package with ported host stack

New workspace package @qwen-code/qwen-live (M1+M2 of the Live split,
issue #10118). This commit adds the package skeleton and the verbatim
ports that need no behavioral changes:

- host/: Live Host protocol v6 types, LiveHostCoordinator, and the
  discovery-file writer, ported from packages/cli/src/serve/live/
  together with their test suites (51 tests green)
- realtime/socket.ts, realtime/sanitize.ts: the two helper symbols the
  realtime client depends on, extracted from cli ui utils
- registered in scripts/build.js buildOrder and root vitest projects

* feat(qwen-live): generalize the realtime client tool surface

Port qwen-realtime-session.ts and adapt it for the orchestration model:
tools and instructions come from config, every declared tool dispatches
through onFunctionCall with the raw argument payload, and outputs are
receipt-style via submitFunctionOutput. The single-handoff machinery
(multiple_handoffs error, steering acknowledgement, open-call complete)
is removed; transport, response arbitration, pendingCalls timing,
barge-in, and defensive validation are unchanged from the port.
capturesTranscript marks handoff-style tools that capture the live
transcript tail at call time.

Test suite adapted: 27 tests green.

* feat(qwen-live): add the BackendAdaptor seam and the qwen serve adaptor

BackendAdaptor is the capability-bit-driven port between the live
orchestrator and a coding-agent backend. QwenCodeAdaptor drives a
running qwen serve daemon through the @qwen-code/sdk DaemonClient:
202-admission prompts, SSE event normalization (chunks and tool calls
live under session_update.data.update), reject-if-idle mid-turn
steering with a plain-prompt fallback, nested permission votes with
voter attribution, and attachment-backed image blocks. 23 unit tests
against an injected fake client.

* feat(qwen-live): orchestrator, tools, injector, permissions, and daemon entry

The M1+M2 core:
- seven receipt-style dispatch tools plus remain_silent, with a
  dispatcher that answers every call (timeout fallback included) and
  session_N/job_N/asset_N handle registries
- LiveSession wires Host audio, the realtime connection, tool handlers
  (appshot via the Host capture channel, handoff with voice-context
  packing and image attachments, busy-session steering), backend event
  pumps, and a bounded graceful-stop drain
- Injector: safe-insertion window (VAD + response-in-flight + playback
  estimate at 48 kB/s with a quiet gap), batch aggregation, per-job
  progress throttling, barge-in drops progress, spoken/detail split
- PermissionBroker: allow_always stays a one-shot protocol vote backed
  by a local TTL rule; remote resolutions retract queued asks
- LiveDaemon: Host WS endpoint (Bearer + Origin-reject), discovery
  publish with fail-fast mutual exclusion and dead-owner handoff,
  appshot channel marked ready (the in-process capture path needs no
  separate verification), SIGINT/SIGTERM shutdown
- SessionLog: JSONL with synchronous rotation (rename-before-reopen)

116 new unit tests; package total 167 green.

* feat(qwen-live): end-to-end test suites against a real qwen serve daemon

Test stack: a real qwen serve (spawned CLI bundle, fake-openai model
endpoint), a scriptable fake DashScope realtime WebSocket server, the
qwen-live daemon as a child process, and a protocol-v6 FakeHost client
— every boundary is the real wire.

Scenarios (11 tests): discovery publish/remove and hello/welcome,
toggle brings up the realtime session (tool schema + instructions
asserted on session.update), two-way audio byte fidelity, the full
handoff loop against real serve with [COMPLETE] injection and spoken
summary, injection-window hold/batch semantics, permission ask/vote
relay, and busy-session steering tiers.

The suites caught a real bug the unit layer could not: per-session
calls must echo the daemon-issued clientId from the create/attach
response (a self-made id is rejected by the client registration guard
and broke handoff admission and vote attribution). QwenCodeAdaptor now
tracks and echoes the issued id per session.

* chore(release): keep @qwen-code/qwen-live versioned independently

Same treatment as sdk/mobile-mcp/node-repl: the incubating package
stays at its own 0.x version instead of lock-stepping with the CLI
release train. The verbatim guard in package-scripts.test.js is
updated alongside.

* fix(qwen-live): harden review findings across dispatcher, injector, and permission relay

Self-review fixes bundled after the adversarial pass:

- adaptor: classify snake_case permission option ids (proceed_once) —
  the word-boundary regex never matched through "_", so an allow vote
  silently degraded to a cancelled outcome and serve dropped the turn
- adaptor: detect the pending-prompt limit by error name as well as
  status, and fall image-bearing steers back to a regular prompt
- dispatcher: a timed-out tool now answers with a pending receipt that
  tells the model not to retry (the handler keeps running)
- injector: permission items flush ahead of the size-capped context
  join; spoken lines merge whole-line greedily under the 280-char cap
- live-session: resubscribe the event pump with backoff until
  session_closed; a rejected permission chain can no longer become an
  unhandled rejection; receipts are clamped and never empty
- permission broker: a failed auto-approval falls back to the spoken
  ask instead of losing the request
- daemon/config/index: appshot readiness is declared at construction,
  numeric config ports parse, stray rejections are logged not fatal
- e2e harness: QWEN_LIVE_E2E_KEEP=1 keeps the temp dirs for post-mortem

* fix(qwen-live): address review round 1 — permission voting, shutdown, discovery reclaim, and hardening

Permission trust boundary:
- classify options by the wire's structured kind enum (allow_once/
  allow_always/reject_*) instead of word regexes over snake_case ids,
  and vote the least-escalating match — a spoken "allow" now lands on
  proceed_once, never proceed_always_project
- respond_permission's note is no longer discarded: it is logged and
  relayed to the backend session as a follow-up user instruction
- track own votes per session so byUs attribution works on adopted
  sessions; narrow allow_always standing rules to the exact normalized
  command instead of a two-token prefix

Lifecycle:
- daemon stop() no longer hangs when SIGINT lands before listen() or
  when a Host redials during shutdown (stopping gate + client terminate)
- dead-owner discovery reclaim now follows the cli contract (no-op
  commitOwner during handoff, republish after) — previously it could
  never succeed and a crash left the daemon unable to start
- stop drain settles on the input-commit ack instead of burning the
  full 30s budget; response.created during the drain no longer flips
  the call back to speaking; a host-initiated start can no longer race
  deactivate()

Robustness:
- a late ASR final for an already-consumed input item is ignored as
  stale_input instead of dropping the call
- raise the Host frame cap to cover a conformant CJK accessibility
  dump (64KiB frame cap contradicted the 32k-char field cap)
- bin entry works through npm's bin symlink (realpath + pathToFileURL)
- steered handoffs that join a running turn reuse the existing job
  instead of orphaning it; over-long steers fall back to a queued
  prompt; session_list unions created cwds, requests a full page, and
  maps hasActivePrompt for untracked sessions; created-session labels
  are applied via session metadata
- injector: jobless throttle keys use the full context, dedup is
  symmetric, and barge-in drops clear the throttle stamp; sentence
  shapers handle CJK punctuation and mid-token periods
- config loader: ENOENT-only swallow, BOM strip, ~ expansion, typed
  port validation, source-accurate errors; session log close() is
  terminal; transcript retention is capped

Tests and docs: new config/daemon/sanitize suites, win32 guards
restored on discovery mode assertions, protocol parity test now checks
both type copies, qwen-live e2e suites join the no-AK PR gate, fake
model gating in the inject e2e matches only the current turn and is
consumed on match, FakeHost answers capture requests, lockfile diff
reduced to qwen-live-only entries, README states the installer gap.

* test(qwen-live): pin injector size caps and steered-message delivery

- injector: 6000-char combined-context cap, whole-line spoken join at
  the 280-char boundary, single-line ellipsis truncation
- steering e2e: assert the joined instruction actually reached the
  model endpoint, not just the receipt wording

* fix(sdk): raise daemon browser bundle budget

Co-authored-by: Qwen-Coder <[email protected]>

* feat(qwen-live): scaffold standalone live daemon package with ported host stack

New workspace package @qwen-code/qwen-live (M1+M2 of the Live split,
issue #10118). This commit adds the package skeleton and the verbatim
ports that need no behavioral changes:

- host/: Live Host protocol v6 types, LiveHostCoordinator, and the
  discovery-file writer, ported from packages/cli/src/serve/live/
  together with their test suites (51 tests green)
- realtime/socket.ts, realtime/sanitize.ts: the two helper symbols the
  realtime client depends on, extracted from cli ui utils
- registered in scripts/build.js buildOrder and root vitest projects

* feat(qwen-live): generalize the realtime client tool surface

Port qwen-realtime-session.ts and adapt it for the orchestration model:
tools and instructions come from config, every declared tool dispatches
through onFunctionCall with the raw argument payload, and outputs are
receipt-style via submitFunctionOutput. The single-handoff machinery
(multiple_handoffs error, steering acknowledgement, open-call complete)
is removed; transport, response arbitration, pendingCalls timing,
barge-in, and defensive validation are unchanged from the port.
capturesTranscript marks handoff-style tools that capture the live
transcript tail at call time.

Test suite adapted: 27 tests green.

* feat(qwen-live): add the BackendAdaptor seam and the qwen serve adaptor

BackendAdaptor is the capability-bit-driven port between the live
orchestrator and a coding-agent backend. QwenCodeAdaptor drives a
running qwen serve daemon through the @qwen-code/sdk DaemonClient:
202-admission prompts, SSE event normalization (chunks and tool calls
live under session_update.data.update), reject-if-idle mid-turn
steering with a plain-prompt fallback, nested permission votes with
voter attribution, and attachment-backed image blocks. 23 unit tests
against an injected fake client.

* feat(qwen-live): orchestrator, tools, injector, permissions, and daemon entry

The M1+M2 core:
- seven receipt-style dispatch tools plus remain_silent, with a
  dispatcher that answers every call (timeout fallback included) and
  session_N/job_N/asset_N handle registries
- LiveSession wires Host audio, the realtime connection, tool handlers
  (appshot via the Host capture channel, handoff with voice-context
  packing and image attachments, busy-session steering), backend event
  pumps, and a bounded graceful-stop drain
- Injector: safe-insertion window (VAD + response-in-flight + playback
  estimate at 48 kB/s with a quiet gap), batch aggregation, per-job
  progress throttling, barge-in drops progress, spoken/detail split
- PermissionBroker: allow_always stays a one-shot protocol vote backed
  by a local TTL rule; remote resolutions retract queued asks
- LiveDaemon: Host WS endpoint (Bearer + Origin-reject), discovery
  publish with fail-fast mutual exclusion and dead-owner handoff,
  appshot channel marked ready (the in-process capture path needs no
  separate verification), SIGINT/SIGTERM shutdown
- SessionLog: JSONL with synchronous rotation (rename-before-reopen)

116 new unit tests; package total 167 green.

* feat(qwen-live): end-to-end test suites against a real qwen serve daemon

Test stack: a real qwen serve (spawned CLI bundle, fake-openai model
endpoint), a scriptable fake DashScope realtime WebSocket server, the
qwen-live daemon as a child process, and a protocol-v6 FakeHost client
— every boundary is the real wire.

Scenarios (11 tests): discovery publish/remove and hello/welcome,
toggle brings up the realtime session (tool schema + instructions
asserted on session.update), two-way audio byte fidelity, the full
handoff loop against real serve with [COMPLETE] injection and spoken
summary, injection-window hold/batch semantics, permission ask/vote
relay, and busy-session steering tiers.

The suites caught a real bug the unit layer could not: per-session
calls must echo the daemon-issued clientId from the create/attach
response (a self-made id is rejected by the client registration guard
and broke handoff admission and vote attribution). QwenCodeAdaptor now
tracks and echoes the issued id per session.

* chore(release): keep @qwen-code/qwen-live versioned independently

Same treatment as sdk/mobile-mcp/node-repl: the incubating package
stays at its own 0.x version instead of lock-stepping with the CLI
release train. The verbatim guard in package-scripts.test.js is
updated alongside.

* fix(qwen-live): harden review findings across dispatcher, injector, and permission relay

Self-review fixes bundled after the adversarial pass:

- adaptor: classify snake_case permission option ids (proceed_once) —
  the word-boundary regex never matched through "_", so an allow vote
  silently degraded to a cancelled outcome and serve dropped the turn
- adaptor: detect the pending-prompt limit by error name as well as
  status, and fall image-bearing steers back to a regular prompt
- dispatcher: a timed-out tool now answers with a pending receipt that
  tells the model not to retry (the handler keeps running)
- injector: permission items flush ahead of the size-capped context
  join; spoken lines merge whole-line greedily under the 280-char cap
- live-session: resubscribe the event pump with backoff until
  session_closed; a rejected permission chain can no longer become an
  unhandled rejection; receipts are clamped and never empty
- permission broker: a failed auto-approval falls back to the spoken
  ask instead of losing the request
- daemon/config/index: appshot readiness is declared at construction,
  numeric config ports parse, stray rejections are logged not fatal
- e2e harness: QWEN_LIVE_E2E_KEEP=1 keeps the temp dirs for post-mortem

* fix(qwen-live): address review round 1 — permission voting, shutdown, discovery reclaim, and hardening

Permission trust boundary:
- classify options by the wire's structured kind enum (allow_once/
  allow_always/reject_*) instead of word regexes over snake_case ids,
  and vote the least-escalating match — a spoken "allow" now lands on
  proceed_once, never proceed_always_project
- respond_permission's note is no longer discarded: it is logged and
  relayed to the backend session as a follow-up user instruction
- track own votes per session so byUs attribution works on adopted
  sessions; narrow allow_always standing rules to the exact normalized
  command instead of a two-token prefix

Lifecycle:
- daemon stop() no longer hangs when SIGINT lands before listen() or
  when a Host redials during shutdown (stopping gate + client terminate)
- dead-owner discovery reclaim now follows the cli contract (no-op
  commitOwner during handoff, republish after) — previously it could
  never succeed and a crash left the daemon unable to start
- stop drain settles on the input-commit ack instead of burning the
  full 30s budget; response.created during the drain no longer flips
  the call back to speaking; a host-initiated start can no longer race
  deactivate()

Robustness:
- a late ASR final for an already-consumed input item is ignored as
  stale_input instead of dropping the call
- raise the Host frame cap to cover a conformant CJK accessibility
  dump (64KiB frame cap contradicted the 32k-char field cap)
- bin entry works through npm's bin symlink (realpath + pathToFileURL)
- steered handoffs that join a running turn reuse the existing job
  instead of orphaning it; over-long steers fall back to a queued
  prompt; session_list unions created cwds, requests a full page, and
  maps hasActivePrompt for untracked sessions; created-session labels
  are applied via session metadata
- injector: jobless throttle keys use the full context, dedup is
  symmetric, and barge-in drops clear the throttle stamp; sentence
  shapers handle CJK punctuation and mid-token periods
- config loader: ENOENT-only swallow, BOM strip, ~ expansion, typed
  port validation, source-accurate errors; session log close() is
  terminal; transcript retention is capped

Tests and docs: new config/daemon/sanitize suites, win32 guards
restored on discovery mode assertions, protocol parity test now checks
both type copies, qwen-live e2e suites join the no-AK PR gate, fake
model gating in the inject e2e matches only the current turn and is
consumed on match, FakeHost answers capture requests, lockfile diff
reduced to qwen-live-only entries, README states the installer gap.

* test(qwen-live): pin injector size caps and steered-message delivery

- injector: 6000-char combined-context cap, whole-line spoken join at
  the 280-char boundary, single-line ellipsis truncation
- steering e2e: assert the joined instruction actually reached the
  model endpoint, not just the receipt wording

* feat(qwen-live): port the Live Host installer — M1 gap closed

`live-host-installer.ts` moves verbatim from
packages/cli/src/serve/live/ (same OSS/GitHub dual-source manifest
fetch, sha256 + codesign + team-identifier verification, staging +
backup install with rollback, and the retryable status machine), with
its 11-test suite. The daemon gains the matching HTTP surface:

- GET /live/setup — installed Host state
- POST /live/setup/install — download + verify + install the latest
  POST /live/setup/launch — open the installed app

All three sit behind the daemon's existing Bearer wall (a browser
context sending Origin is always refused, wrong token 401) — pinned by
a new daemon test; unknown paths stay 404 and GET /healthz answers
plain 200. Bootstrap no longer depends on the built-in qwen-code Live
integration having installed the Host first.

* fix(qwen-live): attach the appshot-timeout rejection handler before advancing fake timers

The timeout rejects synchronously inside advanceTimersByTimeAsync; a
handler attached only afterwards leaves an unhandled rejection for an
instant, which vitest counts as a run-level error (CI fails the
package with Errors: 1 even though all 250 tests pass). Attach the
expectation first, await it after the clock advances.

* test(qwen-live): sync the no-AK gate pin with the qwen-live suites

The no-ak-integration-ci pin (added upstream after our branch point)
asserts the gate's file list verbatim; add the four qwen-live suites
the gate has been running since they joined it.

* fix(qwen-live): address review round 2 — trust-boundary keys, backpressure, session lifecycle

Criticals (all probed and reproduced before fixing):

- broker titleKeyOf: split on the FIRST colon and keep the entire
  remainder (commands routinely contain colons — a truncated key
  auto-approved a different -m message), and stop case-folding
  (/a vs /A are different paths on Linux). The whitespace-collapse
  pin is preserved
- broker deny: an explicit spoken refusal now revokes the standing
  rule that matched the same request — the silent auto-answer's
  fallback path could otherwise be overridden by the invisible rule
  on the next identical request
- adaptor activeJobRef: a busy-session fallback (image steer, over-
  long text 400) no longer overwrites the running turn's ref with the
  queued prompt's id — later steer receipts report the right job
- adaptor byUs attribution: the daemon's originator stamp is now
  authoritative when present; ownVotes only covers anonymous
  resolutions. Another client resolving during our vote round-trip no
  longer reads as our own answer (R1-2 regression, fixed test-first)
- adaptor permission titles: control sequences stripped before the
  title reaches speech or keys a standing rule (whole command kept —
  no first-line cut, unlike the progress path)
- live-session pushAudio: the provider's backpressure signal is
  propagated (the port source fails the call) and onAudioDropped is
  wired to failCall — a socket over its buffer cap no longer silently
  garbles the utterance; the deliberate true for stale/connecting
  frames is untouched
- handles: closed sessions stop resolving entirely (registry-level),
  and session_closed clears the call's default handle so a session-
  less handoff rebuilds instead of retrying a dead target forever
- handles: read-site reconciliation — session_list and session_monitor
  transition a session's stale non-terminal jobs to done when the
  backend itself reports idle (a turn_complete emitted while no pump
  was subscribed no longer reports a running active_job forever)

Suggestions on this PR's new surfaces:

- live-host.yml: packages/qwen-live/src/host/** joins the path filter
  so the parity-test lane runs on qwen-live-only PRs instead of first
  failing in the merge queue
- installer: the install route passes force=true (the recovery path
  for a corrupted-but-present app, matching the CLI mirror), and
  launch() gains the non-darwin guard in BOTH copies so the byte
  parity pin stays green
- index: directory-form invocation (node packages/qwen-live) now
  runs main() instead of exiting 0 silently
- realtime: the dead lastHandoffEntryCount machinery is removed
  (truncate-and-reset replaced the cursor it pretended to track)

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: 易良 <[email protected]>
Co-authored-by: yiliang114 <[email protected]>
Co-authored-by: Qwen-Coder <[email protected]>
2026-09-01 06:37:23 +00:00

162 lines
5.3 KiB
TypeScript

/**
* @license
* Copyright 2026 Qwen
* SPDX-License-Identifier: Apache-2.0
*/
/**
* Short-handle registries. The realtime model refers to sessions as
* `session_N`, jobs as `job_N`, and screen-capture assets as `asset_N` —
* short, speakable, and hard to mistranscribe. These maps own the
* bidirectional translation to the real identities.
*/
import type { BackendHandle } from '../adaptor/types.js';
export interface JobRecord {
jobHandle: string;
sessionHandle: string;
backend: BackendHandle;
/** Adaptor-side correlation id (qwen serve: promptId). */
jobRef?: string;
state: 'accepted' | 'running' | 'done' | 'failed' | 'cancelled';
task: string;
createdAt: number;
}
export interface AssetRecord {
assetHandle: string;
path: string;
mimeType: string;
createdAt: number;
}
const MAX_ASSETS = 32;
export class HandleRegistry {
private readonly sessions = new Map<string, BackendHandle>();
private readonly sessionsByBackendId = new Map<string, string>();
/** Sessions whose backend reported closure — handles no longer resolve. */
private readonly closedSessions = new Set<string>();
private readonly jobs = new Map<string, JobRecord>();
private readonly jobsByRef = new Map<string, string>();
private readonly assets = new Map<string, AssetRecord>();
private sessionSeq = 0;
private jobSeq = 0;
private assetSeq = 0;
/** Register (or return the existing handle for) a backend session. */
session(backend: BackendHandle): string {
const key = `${backend.adaptor}:${backend.id}`;
const existing = this.sessionsByBackendId.get(key);
if (existing) return existing;
const handle = `session_${++this.sessionSeq}`;
this.sessions.set(handle, backend);
this.sessionsByBackendId.set(key, handle);
return handle;
}
resolveSession(handle: string): BackendHandle | undefined {
const key = handle.trim();
if (this.closedSessions.has(key)) return undefined;
return this.sessions.get(key);
}
/** Mark a session closed: its handle stops resolving from now on. */
closeSession(handle: string): void {
this.closedSessions.add(handle.trim());
}
/**
* Register a new job. Idempotent by jobRef: a jobRef already registered
* means the backend attributed the prompt to that existing turn (e.g. a
* steer that joined it), so the existing record is returned instead of
* minting a second job that would orphan the first in 'running'.
*/
createJob(fields: {
sessionHandle: string;
backend: BackendHandle;
jobRef?: string;
task: string;
}): JobRecord {
if (fields.jobRef !== undefined) {
const existing = this.jobByRef(fields.jobRef);
if (existing) return existing;
}
const record: JobRecord = {
jobHandle: `job_${++this.jobSeq}`,
sessionHandle: fields.sessionHandle,
backend: fields.backend,
...(fields.jobRef !== undefined ? { jobRef: fields.jobRef } : {}),
state: 'accepted',
task: fields.task,
createdAt: Date.now(),
};
this.jobs.set(record.jobHandle, record);
if (record.jobRef !== undefined) {
this.jobsByRef.set(record.jobRef, record.jobHandle);
}
return record;
}
resolveJob(handle: string): JobRecord | undefined {
return this.jobs.get(handle.trim());
}
/** Find the job a backend event's jobRef belongs to. */
jobByRef(jobRef: string): JobRecord | undefined {
const handle = this.jobsByRef.get(jobRef);
return handle ? this.jobs.get(handle) : undefined;
}
/**
* Reconcile a session's non-terminal jobs against reality: the backend
* reports idle, so any 'accepted'/'running' record left over from a
* missed terminal event (emitted while no pump was subscribed — pumps
* are per-call and aborted at call end) transitions to 'done'. Gated on
* the caller's isBusy check so a genuinely busy backend is never
* touched.
*/
reconcileIdleSession(sessionHandle: string): void {
for (const job of this.jobs.values()) {
if (job.sessionHandle !== sessionHandle) continue;
if (job.state === 'done' || job.state === 'failed') continue;
if (job.state === 'cancelled') continue;
job.state = 'done';
}
}
/** The most recent non-terminal job for a session, if any. */
activeJobForSession(sessionHandle: string): JobRecord | undefined {
let candidate: JobRecord | undefined;
for (const job of this.jobs.values()) {
if (job.sessionHandle !== sessionHandle) continue;
if (job.state === 'done' || job.state === 'failed') continue;
if (job.state === 'cancelled') continue;
if (!candidate || job.createdAt > candidate.createdAt) candidate = job;
}
return candidate;
}
registerAsset(fields: { path: string; mimeType: string }): AssetRecord {
const record: AssetRecord = {
assetHandle: `asset_${++this.assetSeq}`,
path: fields.path,
mimeType: fields.mimeType,
createdAt: Date.now(),
};
this.assets.set(record.assetHandle, record);
// Bounded: assets reference temp files with short TTLs; keep the newest.
while (this.assets.size > MAX_ASSETS) {
const oldest = this.assets.keys().next().value;
if (oldest === undefined) break;
this.assets.delete(oldest);
}
return record;
}
resolveAsset(handle: string): AssetRecord | undefined {
return this.assets.get(handle.trim());
}
}