2 Commits

Include renames

Author	SHA1	Message	Date
顾盼	a791e47a54	feat(cua): streamline computer use and port native reliability fixes (#14846) ... * feat(cua): preload the JavaScript computer use runtime * refactor(cua): remove the native Wayland backend * feat(cua): move app computer use into the driver * feat(cua): preserve background delivery and port platform fixes * feat(cua): add explicit desktop input and browser runtime facade * feat(cua): package the unified 0.26.0 computer use runtime * fix(cua): finalize cross-platform native reliability * fix(cua): isolate runtime releases from CLI versioning * fix(cua): keep Electron out of root workspace installs * fix(cua): synchronize consumers and async cleanup tests * fix(browser): migrate smoke to preloaded cua runtime * docs(cua): document MCP setup for Qwen Code Claude Code and Codex	2026-10-21 14:03:31 +08:00
8 people	b7543aeb1b	feat(browser-use): add Chrome Native Messaging relay (#11242) ... * feat(browser-use): add Playwright browser SDK * docs(browser-use): document browser use architecture * fix(browser-use): make browser input actions reliable * feat(browser-use): preserve screenshot coordinate metadata * fix(browser-use): recover stale sockets across VM realms * fix(browser-use): bound locator action retries * fix(browser-use): keep page input in the background * fix(browser-use): reject ambiguous input arguments * fix(browser-use): validate locator composition identity * fix(browser-use): recover socket locks across VM realms * fix(browser-use): preserve input results across navigation * fix(browser-use): capture screenshots through bounded CDP frames * fix(browser-use): separate input and navigation waits * feat(browser-use): add Chrome native relay * fix(chrome-extension): avoid foreground tabs and stale cursor markup * fix(chrome-extension): require Chrome 125 for flat sessions * feat(browser-use): add product integration * test(browser-use): add managed Chrome smoke coverage * feat(node-repl): expose structured image metadata * test(browser-use): preserve complete smoke reports * feat(browser-use): bundle skill and runtime with CLI * fix(browser-use): install pinned Playwright in fresh workspaces * feat(browser-use): require explicit skill opt-in * feat(browser-use): make bundled skill available by default Keep explicit skill disables authoritative, update documentation and tests, and use the latest Node REPL package in setup instructions. * fix(ci): build browser-use before Live Host publish Build the bundled browser runtime before core in the Live Host npm dependency chain, and cover that prerequisite order with a regression test. * fix(browser-use): output screenshot metadata explicitly * fix(node-repl): preserve image metadata outside prose budget * fix(browser-use): address review round on workspace wiring and runtime correctness - register the workspace in .pnpmfile.mjs, pnpm-lock.yaml, and the unit-vitest-configs registry, with the shared-pool timeout ternary and off-Linux unhandled-error exemption (fixes the red Install/Test checks) - map Playwright's 'warning' console type to the contract's 'warn' level - reject captures whose pixel dimensions contradict the css-pixels coordinate contract instead of certifying rescaled images - keep one unattachable derived tab from failing tabs.list while still propagating a lost session - clear the cached dialog when Chrome resolves it out of band or the page navigates, guarded by dialog identity so a newer dialog survives - synthesize a parseable Browser.getVersion (numeric Chrome major, host platform token) so Playwright enables the right keyboard editing commands - keep YAML-quoted snapshot keys and link props in interactive listings and stop claiming subtrees that filtering removed - always release the mouse button when a drag gesture fails * fix(dev): build browser-use runtime before startup * fix(browser-use): isolate cleanup and correct smoke checks * fix(browser-use): resolve managed Chrome through Playwright * fix(browser-use): harden native relay and tab lifecycle * fix(browser-use): drain host output and persist restored tab state * fix(browser-use): scope dialog handles and clean navigation waiters * fix(browser-use): preserve input outcomes and release modifiers * fix(browser-use): drain session teardown and release debugger attachments * fix(browser-use): correct locator validation and evaluation * fix(dev): reuse prebuilt Browser Use runtime * fix(browser-use): address review round on locator, bridge, and screenshot correctness Co-authored-by: Qwen-Coder <[email protected]> * fix(browser-use): require installed extension before native host setup * fix(browser-use): address review round on schema probe bound and regex validation * fix(browser-use): start transport before Chrome preflight * fix(browser-use): exclude staged runtime from code checks * fix(core): declare Browser Use build dependency * fix(browser-use): preserve successful navigation and dialog input * fix(browser-use): harden input drain and tab registration cleanup * fix(browser-use): keep sessions bound to one Chrome profile * fix(browser-use): bound commands and validate evaluation JSON * fix(browser-use): reuse unchanged Native Host registration * fix(browser-use): wait for fresh extension installation * fix(browser-use): address review round on screenshot origin, dialog scope, and bridge privacy - screenshot: tolerate settle-probe failure (measure the ratio from the captured pixels, re-capture on HiDPI), measure layout after the settle wait, and reject screencast frames whose scroll offsets disagree with the published origin - session: clear the cached dialog only on main-frame navigation; abort tab attachment when the tab is removed mid-attach - errors: classify crashes and timeouts by Playwright's structured error name instead of page-influenced message text - locator.press: release chord modifiers when a later token is rejected; cap locator.type at the 60k characters its per-character budget can deliver inside the 120s ceiling - snapshot: keep AI-mode [cursor=pointer] generic nodes in the interactive listing - bridge: derive the fallback socket directory as an env-independent per-user 0700 subdirectory on all POSIX hosts and verify ownership before binding * fix(browser-use): close review-critical gaps in error mapping, budgets, and downloads - errors: classify by Playwright error name first; text phrases match only the start of the first message line so page-controlled text (log tails, quoted markup, page-thrown values) cannot pick the reported code, and a TimeoutError name wins over crash-shaped page text - transport: cdp.send bridge requests default to 130s so operations up to the 120s schema ceiling report their own deadline; other requests keep the 30s default - transport: the /tmp fallback socket directory is a per-user leaf directly under the sticky root (no shared intermediate a first mover could own); ensureSocketDirectory re-verifies the leaf after creation and walks ancestors up to the sticky root, requiring each to be owned by this user or root and not group/other-writable - runtime: withModifiers no longer rewrites a completed action into a failure when a modifier release fails; chord failure cleanup releases the chord's own tokens (including non-modifiers) instead of four fixed names - playwright: enable CDP download events on tab attach (Playwright's noDefaults connect leaves them off, so waitForEvent('download') could never resolve); locator.downloadMedia races the caller deadline, aborts the in-page fetch at it, resolves media contained in a located wrapper before an ancestor link, restricts URL schemes, and names CORS rejections distinctly; locator.allTextContents waits for the first match within the caller budget and still resolves [] when nothing attaches - screenshot: full-page captures get a 16 Mi-pixel budget (1920x3000 pages now succeed) and the encoded byte ceiling drops to the 4 MiB model image limit so captures are not dropped at the model boundary - snapshot: interactiveOnly promotes only a terminal [cursor=pointer] attribute (page text quoting the marker no longer promotes the line), and truncation skips over-long lines instead of discarding the rest - schemas: reject empty string/regex matchers and exact:true combined with a regex matcher (Playwright silently drops exact for regexes) - tests: the SDK evaluate fixture now mirrors production serialization; POSIX socket tests carry win32 gates; posix.join keeps the derived socket path a pure function of uid and platform on every host * fix(browser-use): close review criticals in error mapping, chords, downloads, and snapshots - errors: classify Playwright's genuine strict-mode violation, which crosses CDP as the raw exception description and reaches the client behind an "Error: " layer ("locator.click: Error: strict mode violation: ... resolved to N elements:"); requiring the element count keeps a page-thrown lookalike from forging LOCATOR_NOT_UNIQUE - runtime-helpers: on a failed chord press, release the tokens Playwright actually derived ('+' separates only after a non-empty token, so a standalone '+' is a literal key) rather than the caller's array elements; an element embedding '+' previously released nothing and left modifiers held. locator.press uses the same tokenizer - locator.downloadMedia: resolve an ordered candidate list (element, media descendants, anchor descendants, closest ancestor) and keep the first candidate that exposes a URL, so a wrapping anchor no longer beats the image it contains and a source srcset or an unloaded video's source child can supply the URL - snapshot: unwrap Playwright's single-quoted YAML key before the [cursor=pointer] suffix test, so clickable nodes whose accessible name forces key quoting stay in the interactive listing - docs: align the Input completion contract with the shipped modifier cleanup semantics (cleanup failures are discarded; only the action's own error propagates) in both languages * fix(browser-use): close round-13 review criticals in errors, srcset, evaluateAll, snapshots, and screenshot origin * fix(browser-use): point the pnpm importer at a vitest snapshot that exists Merging main brought in #11625, which regenerated pnpm-lock.yaml and dropped the [email protected] snapshot keyed with @vitest/browser and [email protected]. The packages/browser-use importer still referenced that key, so every Install leg failed with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY. Re-key the importer to the snapshot its sibling packages already use; nothing else in the lockfile changes. * fix(browser-use): close round-14 criticals in error classification, dialog ordering and screenshot ratio R13-2: the strict-mode branch of sanitizeOperationError was the only text-phrase branch not gated on the evaluate channel, so a page-thrown primitive "strict mode violation: ... resolved to 2 elements:" picked LOCATOR_NOT_UNIQUE. On that channel only Playwright's "Error: "-wrapped form classifies now; other channels keep the unwrapped form. R14-1: Playwright hands a dialog over on a later turn than the bridge reports its CDP events (a microtask in CRSession plus the in-process dispatcher's setImmediate), so an open+close pair delivered in one chunk left a phantom dialog that blocked the tab. Each tab now traces dialog openings and closes in bridge order and drops a delivered dialog the bridge already reported closed; a close with no traced opening (a dialog open before the tab was attached) is never charged to a later dialog, and the microtask clear stays for the delivered case. R14-2: window.devicePixelRatio is page script, and a spoofed value became an unbounded clip.scale while the pixel re-measure only ran when the probe failed. The probed ratio is now trusted only within 0.25..8, and the measure-and-recapture correction runs whenever the first fallback capture disagrees with the requested CSS region, deriving the real ratio from Chrome's own pixels (measured / (width * scale)). Also resolve the temp root in the ensureSocketDirectory tests: the ancestor sweep rejects symlinks, and macOS reaches os.tmpdir() through /var -> /private/var, so the tests failed locally on macOS. * fix(browser-use): tear down accepted peers on start failure and pin the transport's untested paths The esbuild banner declared require/__filename/__dirname/process bindings the bundle never uses, so any bundled module declaring one of them built green but failed to load; drop it (and the unused .wasm loader) and import dist/index.js after bundling so an unloadable bundle fails the build. When start() fails after listen (chmod, handshake), a peer validated in between kept server.close() waiting forever; destroy and clear the accepted sockets first, mirroring stop(). net.Server has no closeAllConnections(), so that is the whole fix. Tests now cover the QWEN_BROWSER_USE_SOCKET_PATH override, a hello with a mismatched protocol version or extension id, onConnectionChange on the real transport, an in-flight request when the extension disconnects and the configured request budget, the ok:false error-code mapping, FrameDecoder split-frame reassembly and its length ceiling, the start-failure teardown, and all five schema refine predicates. * fix(browser-use): mirror the contract bounds in the SDK and state the documented limits nth() now rejects indexes outside the contract's -10000..10000 before dispatch, and all() fails closed when the plan already has 32 steps or the match count exceeds what nth() can address, instead of handing back locators that only fail on first use. Both evaluate script builders normalise method-shorthand and class-method sources the way Playwright does (compile-only vm.Script probe, then a function prefix), so those well-typed callbacks no longer die as a page-side SyntaxError. providerTab clamps title and url to the claimTab schema's 20000-char cap so the documented openTabs() -> claimTab() round trip survives long URLs, and orderOpenTabs() establishes the http(s)-only, newest-first listing the model documentation promises. downloadMedia derives the file name from the URL path only for http(s) sources and leaves blob:/data: names to the browser. The model-facing documentation now states the 1s read default, that dom_cua.scroll's x/y are wheel deltas rather than coordinates, and that a cross-origin resource without CORS headers cannot be downloaded from the page origin; the design docs record that boundary as follow-up work. Tests pin every one of these, plus buildLocator's frame, combinator, filter and positional branches, the file-chooser SDK surface, and the claimTab argument guards. * fix(browser-use): keep background tab registration from moving the selection registerTab selected every tab it registered, and two of its callers are background paths (derivedTabTracked, syncDerivedTabs via tabs.list), so a read-only tabs.list could move tabs.selected onto a popup. Selection is now set only by newTab and claimTab from the info they return. Also: tabs.get is allowed while a dialog is open, matching tabs.list and tabs.selected, which already return the dialog-safe TabInfo; the auxiliary-button click moves Playwright's tracked cursor before sending the CDP button events so a later wheel lands where the click did; the dead providerTabId lookup is removed; openTabs applies orderOpenTabs. Tests: the runtime test file mocks node:timers so vi.useFakeTimers() actually controls the input drain bound (the 250ms test drops to ~0ms), and new cases pin the drain clearTimeout, the registration probe's session detach, promptText pass-through, tabs.get under a dialog, the openTabs/claimTab fail-closed guards and ordering, popup adoption from derivedTabTracked, the abandoned navigation waiter's swallowed rejection, the uploadFiles path guards, and the screencast fallback and byte budget on the unconstrained capture path. * test(browser-use): pin the element bindings the runtime gives evaluate scripts The SDK emits scripts that read the free variables `element` and `elements`, and the runtime binds them when it compiles the script inside the page; no test ran either page-side callback, so swapping the binding name on one side kept the suite green while every locator.evaluate would throw a ReferenceError in production. Run the real callbacks with fake elements so the contract is pinned from both ends. * fix(browser-use): fail closed on page-controlled strict-mode text and prefer contained media over anchor hrefs * fix(browser-use): bound page title reads and download a located anchor's file over its icon * fix(browser-use): correct media downloads and frame composition * fix(browser-use): bound locator.count and allTextContents by the caller deadline * fix(browser-use): preserve native snapshots for DOM CUA Co-authored-by: Qwen-Coder <[email protected]> * fix(browser-use): classify a closed tab from the tab's own state The pinned playwright-core client renders a closed target as a plain Error: only TimeoutError sets `name`, so the `name === 'TargetClosedError'` branch in sanitizeOperationError never fired, and because evaluate-channel text fails closed by design, a tab that closed or crashed while a page command ran was reported as OPERATION_FAILED instead of the documented STALE_TAB. Decide tab-gone in the dispatcher from `tab.stale` and `page.isClosed()`, which the page observers keep current, before the text classifier runs. Drop the dead name branch and replace the test that fabricated the name with one built the way the client builds the error, plus dispatch-level cases for both signals that fail without the dispatcher check. Co-Authored-By: Claude Fable 5.1 <[email protected]> * fix(browser-use): restore the kernel realm process binding and land the round-5 review batch Criticals from review round 5, each reproduced before fixing: - The bundled skill imports the staged runtime inside the node_repl kernel, whose untrusted vm context has no `process` global and denies the `node:process` specifier. 3fc2dc3e10 removed the bundle banner that bound `process` through createRequire, so the skill's first cell threw `ReferenceError: process is not defined`; the build's own load check runs in the main realm and could not see it. Restore the binding on the index build and add a kernel-backed test that runs SKILL.md's first cell through the real kernel and fails on the ReferenceError (R5-4). - Bump @qwen-code/node-repl-mcp to 0.1.6: this PR changes its published source (image metadata channel) and the package is excluded from the repo-wide bump, so the release lane's integrity check would fail. Pin the browser-use skill to that version, have the skill test assert the pin equals the package version, and extend the cua-driver release gate to cross-check the browser-use pin (R5-2, R3-5). - Report an incompatible extension as EXTENSION_VERSION_MISMATCH so browser discovery surfaces the update guidance instead of an empty list after the connect budget (R5-5). - Make tab grouping non-fatal in tabs.create, matching its siblings, so a cosmetic grouping failure no longer closes the tab just created (R5-6). Review suggestions verified still applying, with tests proven red under mutation: typecheck the scripts directory; pin the Native Messaging chunk boundary and envelope; return installedPaths from the installer; treat a blank install-home override as unset; warn on a skipped foreign manifest; refuse a foreign launcher with the refusals' missing red arms; keep the preflight download waiter handled; settle the parked-profile test; name the private-directory cause in the bridge start error; declare the finalize status union once; reject unknown or closed kept tabs; pin the extension's protocol constants to the SDK source; resolve the extension test root from the module; drop the duplicated permission assertion; make the group barrier real; read the overlay style rules; map a vanished tab to STALE_TAB in the relay; clamp listed tab text to the schema bound; remove the unreachable onDetach child-session branch; assert the raced tab in the disconnect test; cover the history scheme filter and session-name reset; and fix the extension-source and prerequisite wording in the docs. Co-Authored-By: Claude Fable 5.1 <[email protected]> * test(browser-use): let the SauceDemo smoke recognize equivalent skill usage The model-driven smoke completed the task correctly on every run but its transcript checks asserted one exact spelling: the literal runtime entry path in the model's code, and one parent-scoped price selector with an exact number of dollar values in the cell's output. A model that builds the entry from the skill root across cells, scopes the price locator differently, or echoes its chosen items after the read failed the smoke. Key the import check on what every spelling shares: the built-in skill root the CLI substitutes into SKILL.md, the entry file name and the setup call. Match price reads on the price class, prefer a read that printed exactly the expected count, take the leading set otherwise, and take the last such read for the cart since the flow reads it after the inventory. The smoke-checks test covers all three import spellings, a foreign runtime and a missing setup call, and the parser's scoped, echoed and non-list reads. Verified against a real Chromium 151 run: all six checks true. Co-Authored-By: Claude Fable 5.1 <[email protected]> * fix(browser-use): close six fail-open paths in the playwright adapter Round-22 review found six Critical defects in the new package; each is fixed at its root and pinned by a test that goes red without the fix: - locator.evaluateAll opened a second full timeout window after the attach wait had consumed the caller's budget; the wait and the read now share one deadline, and a wait that consumed the budget settles [] without evaluating. - tab.goto passed any string to page.goto, so a file:// URL would have handed local file contents to the tab's evaluate and snapshot reads; goto is gated to http(s) before the navigation runs. - Synthesized CDP error responses carried no code, so a reply orphaned by a target crash tripped Playwright's internal assert and tore down the whole transport; they now carry -32001, the orphan-swallow code. - A dialog already open when a tab attaches is replayed by neither Chrome nor Playwright, so the DIALOG_OPEN gate stayed blind while the modal blocked the renderer; registration now probes the renderer and gates the tab on an unanswered probe, with getJsDialog reporting it under a stable handle whose accept/dismiss run over CDP. - A dialog opening Playwright never delivered stayed in the trace forever and swallowed the charge of every later dialog on the tab; a closed, undelivered entry is retired once the dispatcher turn has passed. - snapshotRefLocator validated only the ref's shape, so a ref from a pre-navigation snapshot could act on whichever element the new document happened to hand the same string; refs are bound to the snapshot that emitted them and cleared on main-frame navigation. Co-Authored-By: Qwen-Coder <[email protected]> * build: exempt the staged Browser Use skill runtime from the dependency-direction check The browser-use package build stages the bundled skill's runtime, node_modules included, under packages/core/src/skills/bundled/browser-use/runtime (git-ignored), and the TUI dependency-direction gate fails on any node_modules inside a protected root. Declare that staged, ignored root as a build artifact so the gate skips it without a diagnostic; every other skipped-directory name still fails. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MWe8PKNMpsKdQruTWkvFFc * build: leave the staged Browser Use runtime out of the dependency-direction walk entirely The staged root is a build artifact, not source: skipping only its node_modules while still parsing the SDK bundle next to it was inconsistent. Do not walk anything under a declared staged root; every other node_modules inside the protected roots remains a diagnostic. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MWe8PKNMpsKdQruTWkvFFc * build(browser-use): stage the skill runtime only for the bundle and for dev mode The package build used to copy the bundled skill runtime, node_modules included, into packages/core/src, which every built tree then carried and which the TUI dependency-direction gate rightly rejects. Follow the qc-helper convention instead: the bundle step stages the runtime into dist/bundled for the shipped CLI, and `npm run dev` stages the git-ignored development copy next to the source SKILL.md, because only dev mode reads bundled skills from the source tree. No build step writes into src any more, so the gate exemption added earlier is dropped and the check script returns to main's version. The kernel-realm test stages its own copy into a temporary skill directory rather than depending on a build side effect. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MWe8PKNMpsKdQruTWkvFFc * fix(browser-use): deliver bridge messages in arrival order and keep dev info flags fast Three fixes from the round-3/4 verification and review round 7–8: - Transport: a response only settles a promise, so its consumer ran on the microtask queue while events from the same socket chunk were emitted synchronously and overtook it. Playwright installs a page's renderer and lifecycle listeners inside `Page.getFrameTree().then(...)`, so an overtaking `Runtime.executionContextCreated` found no listener and a claimed tab lost its main world for good (B1). Inbound messages now go through one per-socket queue whose drain yields a macrotask after settling a response, so every continuation runs before the next message; chunks arriving meanwhile join the same queue. Regression: `delivers a response ahead of the events that share its socket chunk`. - Relay: `requireControllableTab` judged a popup by `tab.url`, which is empty until the navigation commits, while the claim keyed on `pendingUrl`; the runtime's immediate `tabs.get` therefore rejected the tab the relay had just reported (R8-2). It now judges by `pendingUrl || url`, like the claim. Regression: `a popup known only by its pending URL stays controllable`. - dev launcher: the dev-mode runtime staging skips `--version`/`--help` (and `-v`/`-h`), restoring the fast path the existing tests pin (R7-1). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MWe8PKNMpsKdQruTWkvFFc * test(browser-use): make the smoke evidence and the persist-before-connect pin honest Round 6–7 review findings on the verification tooling: - The SauceDemo smoke decided "the staged runtime was loaded" from three substring tests over all cells joined, so a run whose staged import threw and fell back to a foreign entry still passed (R6-1). It now looks at the import expressions per cell: a literal path under the staged runtime, or a path assembled from the substituted skill root, in a cell that reaches the setup export; a literal entry loaded from anywhere else disqualifies the run. - Price reads were attributed to the cart by count and read order, so an inventory sorted low-to-high passed as a cart read and the fallback to the first read never returned null (R6-2). Reads are now attributed by the tab.url() confirmation of cart.html the prompt requires: inventory before it, cart after it, and no cart read without it. - The persist-before-connect invariant was asserted inside the mock's postMessage, which the bridge calls under a bare catch, so the reordered mutant stayed green (R7-2). The mock now records every hello with the identity persisted at that moment and the test asserts the order outside the bridge; the mutant is red. - SKILL.md states the node-repl-mcp 0.1.6 requirement next to the emitImage step the agent always reads, with how to detect and re-register an older server (R6-3). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MWe8PKNMpsKdQruTWkvFFc --------- Co-authored-by: qwen-code-dev-bot <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Qwen-Coder <[email protected]> Co-authored-by: qwen-code-dev-bot <[email protected]> Co-authored-by: Shaojin Wen <[email protected]> Co-authored-by: qwen-code-ci-bot <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]>	2026-09-18 05:50:24 +00:00