mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
fix(cua): bound Linux observations and prepare SDK 0.20.9 (#11910)
* fix(cua): prevent Linux input replay and redundant observations * fix(cua): bound Linux accessibility collection for SDK 0.20.9
This commit is contained in:
28 files changed
+1895
-442
No files matched your search
@@ -30,7 +30,7 @@ on:
|
||||
version:
|
||||
description: 'Version to release (without v prefix)'
|
||||
required: true
|
||||
default: '0.20.8'
|
||||
default: '0.20.9'
|
||||
notarize:
|
||||
description:
|
||||
'Codesign + notarize the macOS artifacts (false to skip during
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# Linux CUA input recovery
|
||||
|
||||
[简体中文](linux-cua-input-recovery.zh-CN.md)
|
||||
|
||||
Linux's semantic-input fallback currently treats failure after dispatch as if
|
||||
the semantic route were unavailable. A native action can mutate the application
|
||||
and then lose its D-Bus reply; the click tool subsequently sends a pixel click.
|
||||
The same fallback exists for scrolling. Some click paths also ignore a false
|
||||
AT-SPI acknowledgement and report success.
|
||||
|
||||
The reproduction uses the released 0.20.8 source: an action effect followed by
|
||||
an error or the existing 25-second timeout produces two fixture effects, one
|
||||
semantic and one pixel. A false acknowledgement with no effect still returns
|
||||
"Clicked element". These are source-extraction reproductions with controlled
|
||||
transport endpoints; they are not claims about the causes of every historical
|
||||
zero-score task.
|
||||
|
||||
Port the macOS recovery principle into the existing Linux implementation:
|
||||
fallback is possible before input dispatch, but a possible dispatch must stop
|
||||
automatic replay. Track dispatch locally in each AT-SPI attempt, including
|
||||
timeouts and partially completed scroll batches. Return an unconfirmed-input
|
||||
error after false, failed, or timed-out acknowledgements. The caller must
|
||||
observe before deciding whether to retry. Preserve safe pre-dispatch fallback,
|
||||
exact target identity and the current foreground guard.
|
||||
|
||||
The existing Node REPL → Computer Use facade → native Linux driver topology,
|
||||
processes, installation and permissions remain unchanged. No new public API,
|
||||
service, global retry policy or provider-specific workaround is introduced.
|
||||
The canonical fixed Computer Use skill remains the model guidance source.
|
||||
|
||||
The same dispatch boundary applies to text insertion and value replacement.
|
||||
An uncertain insert must not fall through to a whole-field replacement, and a
|
||||
rejected replacement must not fall through to an append. Successful writes
|
||||
without readback remain unverified; they must not automatically emit a
|
||||
foreground escalation that the shared core interprets as failed delivery.
|
||||
|
||||
Preserve editable text independently of the accessible label, including an
|
||||
empty value when clearing a field. Compact observation rows omit duplicate
|
||||
label/value text, default states and the primary activation action, while the
|
||||
structured elements retain all actions. Keep nondefault state and secondary
|
||||
actions visible so actionable differences survive revision comparison.
|
||||
|
||||
Pixel fallbacks retain the exact window and element identity. Intersect the
|
||||
element bounds with that window and recompute the point after foreground
|
||||
preparation, preventing oversized controls from targeting a sibling window.
|
||||
GTK3 already supplies correct screen coordinates with client-side decorations;
|
||||
avoid applying GTK4's shadow reconstruction to that case. Preserve and verify
|
||||
the GTK4 path separately. Its X11 top-level Screen extents are window-local,
|
||||
so correlate a multi-window GTK4 application only through titles that are
|
||||
unique in both its native and accessibility window lists. Missing or duplicate
|
||||
titles continue to refuse. Semantic primary activation applies only to an
|
||||
unmodified single left click, preserving requested right and double clicks.
|
||||
|
||||
The owned Linux desktop also reproduced active-window and core-focus readiness
|
||||
taking 644–734 ms. Allow one second for the existing focus confirmation loop,
|
||||
which still returns immediately when ready and retries only preparation.
|
||||
The action callback remains single-shot; this does not replay uncertain input
|
||||
or establish the cause of historical benchmark focus failures.
|
||||
|
||||
Linux continues to use its existing exact-window public API. Port only the
|
||||
applicable macOS batching guidance into the canonical skill: combine known
|
||||
actions and emit final verification with cleanup in one cell, while observing
|
||||
new dialogs and menus before acting.
|
||||
|
||||
Observation retries also share one native 25-second budget. Previously four
|
||||
native attempts each received a fresh budget; the SDK's existing read-failure
|
||||
retry could then repeat all four. Retry complete root-only or empty trees for
|
||||
lazy toolkit registration, but return failed or incomplete reads immediately.
|
||||
Preserve the native partial tree and diagnostic, or the X11 fallback when no
|
||||
tree exists. Direct native consumers receive the first failure rather than
|
||||
implicitly retrying it. The SDK keeps one retry for short read failures, but
|
||||
returns deadline-exhausted partial captures without starting another full walk.
|
||||
The companion [observation design](linux-observation-memory-and-compaction.md)
|
||||
covers structural compaction and bounded LibreOffice collection.
|
||||
|
||||
## Validation
|
||||
|
||||
Reproduce the released code before editing. Verify true/false acknowledgement,
|
||||
failure before dispatch, failure after an externally observed effect, timeout,
|
||||
and partial scroll progress. Assert actual effect counts and absence of a
|
||||
second actuator, alongside the model-visible error. Exercise element and
|
||||
coordinate routes and preserve exact native identity on fallback. Then repeat
|
||||
MCP → Node REPL → SDK input, focus restoration and screenshot tests on the
|
||||
owned Linux VM, with independent application readback.
|
||||
|
||||
Compare right and double clicks against coordinate controls. Verify named text
|
||||
set, insertion and clearing through both the observation and an independent
|
||||
application state file. Exercise oversized controls, two windows from one
|
||||
process, GTK3 decorations and GTK4 geometry separately. These GUI checks do not
|
||||
prove transport-failure semantics: the GTK accessibility bridge acknowledges
|
||||
its queued callback independently of that callback's return value.
|
||||
|
||||
Exercise complete and incomplete trees, unavailable and failed reads, cold
|
||||
registration recovery, and budget exhaustion. Check both the native wrapper
|
||||
and actual SDK facade with counted endpoints. Verify that the last valid
|
||||
minimal tree survives retry exhaustion and each attempt receives only the
|
||||
remaining budget. Measure actual model waits separately from allocated native
|
||||
time budgets and provider latency.
|
||||
|
||||
Historical task terminal status, evaluator score, unique model responses,
|
||||
uncached/cached input and output tokens are separate metrics. Fewer native
|
||||
calls or text characters do not establish lower model cost. Paired real-task
|
||||
evidence is required before claiming an efficiency or score improvement.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Linux CUA 输入恢复
|
||||
|
||||
[English](linux-cua-input-recovery.md)
|
||||
|
||||
Linux 的语义输入回退此前会将分发后的失败视为语义通道不可用。原生动作可能已经
|
||||
修改应用状态，随后丢失 D-Bus 回应，而点击工具继续发送像素点击。滚动也存在相同
|
||||
回退；部分点击路径还会忽略 AT-SPI 的 false 确认并报告成功。
|
||||
|
||||
使用已发布的 0.20.8 源码复现：动作产生效果后返回错误或触发原有 25 秒超时，
|
||||
夹具会记录两次效果，分别来自语义动作和像素动作。未产生效果且返回 false 的确认
|
||||
仍被报告为 “Clicked element”。这是使用受控传输端点、提取实际源码的复现，
|
||||
不能据此认定所有历史零分任务的原因。
|
||||
|
||||
将 macOS 的恢复原则迁移到现有 Linux 实现：输入分发之前可以回退，一旦可能已经
|
||||
分发，就必须停止自动重放。每次 AT-SPI 尝试局部记录分发状态，覆盖超时和只完成
|
||||
部分步骤的滚动批次。确认返回 false、失败或超时时，返回输入结果未确认错误；
|
||||
调用方必须先观察，再决定是否重试。保留安全的分发前回退、精确目标身份和现有
|
||||
前台保护。
|
||||
|
||||
Node REPL → Computer Use facade → Linux 原生驱动的现有调用结构、进程、安装和
|
||||
权限保持不变。不引入新的公共 API、服务、全局重试策略或特定 provider 的绕过。
|
||||
模型指引仍来自固定的 Computer Use 主 skill 文件。
|
||||
|
||||
文本插入和整体值替换采用同一分发边界。结果不确定的插入不能继续回退为整字段
|
||||
替换，被拒绝的替换也不能回退为追加。成功写入但没有回读时仍属于未验证结果；
|
||||
不能自动报告前台升级要求，否则共享 core 会把它解释为投递失败。
|
||||
|
||||
可编辑文本独立于 accessible label 保存，清空字段时也保留空值。紧凑观察行省略
|
||||
重复标签和值、默认状态及主激活动作，结构化元素仍保留全部动作。非默认状态和
|
||||
次要动作继续可见，确保 revision 比较保留有意义的操作差异。
|
||||
|
||||
像素回退保留精确窗口与元素身份。元素边界与窗口取交集，并在前台准备后重新计算
|
||||
点击点，避免过大的控件误点同进程的其他窗口。GTK3 已提供含客户端装饰的正确屏幕
|
||||
坐标，不再套用 GTK4 的阴影重建。GTK4 路径单独保留并验证：其 X11 顶层 Screen
|
||||
范围实际上是窗口局部坐标，因此同进程多窗口只能通过在原生窗口列表和可访问性
|
||||
窗口列表中均唯一的标题关联。缺失或重复标题继续拒绝。语义主激活仅适用于无修饰
|
||||
键的单次左击，保留用户请求的右击与双击语义。
|
||||
|
||||
独占 Linux 桌面还复现了活动窗口与 core 焦点就绪需要 644–734 毫秒。将现有焦点
|
||||
确认循环的预算设为一秒；就绪后立即返回，只重试准备过程。动作回调仍只执行一次，
|
||||
不会重放结果不确定的输入，也不能据此认定历史 benchmark 的焦点失败原因。
|
||||
|
||||
Linux 继续使用精确窗口公共 API。仅将适用的 macOS 批处理指引加入固定 skill：
|
||||
合并已知动作，在一个 cell 中完成最终验证和清理，遇到新对话框或菜单时先观察。
|
||||
|
||||
观察重试共享一个原生 25 秒预算。此前四次原生尝试各自获得完整预算，SDK 的
|
||||
读取失败重试还可能再执行一轮。为处理工具包延迟注册，只对完整但为空或仅有根
|
||||
节点的树重试；失败或不完整读取立即返回。保留原生部分树和诊断，无树时保留
|
||||
X11 回退。直接原生调用方收到第一次失败，不再隐式重试。SDK 对短暂读取失败
|
||||
仍保留一次重试，但耗尽截止时间的部分捕获直接返回，不再启动完整遍历。
|
||||
配套的[观察设计](linux-observation-memory-and-compaction.zh-CN.md)说明结构
|
||||
紧凑化和 LibreOffice 有界采集。
|
||||
|
||||
## 验证
|
||||
|
||||
修改前先复现已发布代码。覆盖 true/false 确认、分发前失败、独立观察到效果后的
|
||||
失败、超时和滚动部分进度。除模型可见错误外，还断言真实效果次数和第二执行器
|
||||
未触发。覆盖元素与坐标路径，确认回退保留精确原生身份。随后在独占 Linux VM
|
||||
重跑 MCP → Node REPL → SDK 输入、焦点恢复和截图测试，并独立回读应用状态。
|
||||
|
||||
将右击和双击与坐标控制组比较。通过观察结果及独立应用状态文件验证有标签文本框
|
||||
的设置、插入、清空。分别覆盖超大控件、同进程双窗口、GTK3 装饰和 GTK4 几何。
|
||||
这些 GUI 验证不能证明传输故障语义：GTK 可访问性桥会在回调排队后确认，不依赖
|
||||
回调本身的返回值。
|
||||
|
||||
覆盖完整与不完整树、不可用与失败读取、延迟注册恢复及预算耗尽。用计数端点验证
|
||||
原生包装和实际 SDK facade。确认重试耗尽后保留最后一份有效最小树，每次尝试
|
||||
只能使用剩余预算。模型实际等待、分配的原生时间预算和 provider 延迟分别度量。
|
||||
|
||||
历史任务终态、评测得分、唯一模型响应数、未缓存/缓存输入和输出 token 是独立
|
||||
指标。原生调用数或文本字符数减少不能证明模型成本下降；宣称效率或得分提升前
|
||||
必须取得成对真实任务证据。
|
||||
@@ -0,0 +1,144 @@
|
||||
# Linux observation memory and compaction
|
||||
|
||||
[简体中文](linux-observation-memory-and-compaction.zh-CN.md)
|
||||
|
||||
## Scope
|
||||
|
||||
Extend the existing Linux recovery candidate in exactly two areas: compact
|
||||
accessibility presentation and bounded LibreOffice observation resource use.
|
||||
App APIs, clipboard and OSWorld code are outside this change. Prepare SDK/driver
|
||||
0.20.9 with synchronized version pins; publishing remains a separate action.
|
||||
The SDK-to-native AT-SPI topology, permissions and installation remain unchanged.
|
||||
|
||||
## Compact presentation
|
||||
|
||||
The previous candidate shortened individual rows, but did not migrate the
|
||||
structural projection present in macOS 0.20.6. Linux also built revision text
|
||||
from actionable nodes alone, losing passive status and instruction text.
|
||||
|
||||
Retain observed passive context, then flatten empty layout containers, remove
|
||||
duplicate parent/child labels, and merge adjacent passive text under the same
|
||||
parent. Preserve window, dialog, menu, list, table and document boundaries,
|
||||
web-content boundaries, focus, selection, disabled state and all actionable
|
||||
nodes. Use iterative projection so deep trees do not consume the call stack.
|
||||
Native indices, identities and the structured action list remain intact.
|
||||
Revision comparison includes the retained static text and editable empty values.
|
||||
Apply projection at the model observation boundary; internal browser setup and
|
||||
consent matching retain original static-label boundaries.
|
||||
|
||||
## Memory investigation
|
||||
|
||||
The original Task035 workbook exposes 1,073,741,824 table children and the
|
||||
`manages-descendants` state. A controlled single `GetChildren` request increased
|
||||
LibreOffice RSS by 107.75 MiB in 24.14 seconds (4.45 MiB/s), while the requesting
|
||||
process stayed flat. The supervisor stopped the owned process at its soft limit;
|
||||
no OOM was needed to reproduce the growth. This is a direct AT-SPI reproduction,
|
||||
not proof of the historical sandbox reboot cause.
|
||||
|
||||
The instrumented b30 candidate also reproduced growth through real SDK calls:
|
||||
all three observations returned X11 fallback, while the trace stopped at the
|
||||
table GetChildren with no reply. During the third call LibreOffice grew at
|
||||
4.52 MiB/s and Node RSS stayed flat. Returning from the caller does not cancel
|
||||
provider-side enumeration.
|
||||
|
||||
For ordinary nodes, replace whole-child arrays with `ChildCount` and lazy
|
||||
`GetChildAtIndex`. Keep one next-sibling cursor per ancestor; check node, depth
|
||||
and time limits before requesting a child. Preserve the existing preorder and
|
||||
shared actionable-index calculation in snapshots and actuators. A depth limit
|
||||
reads metadata without expanding children; a failed child request marks the capture incomplete. The fix does not
|
||||
disable Calc accessibility or allocate a complete array and truncate afterward.
|
||||
|
||||
Indexed access alone proved insufficient: three passes over the same 512 real
|
||||
cells each completed in about 2.8 seconds, but each retained another 3.3–3.4 MiB
|
||||
in LibreOffice and some cells received new AT-SPI object paths on every pass.
|
||||
AT-SPI explicitly says clients should not enumerate `ManagesDescendants` nodes.
|
||||
Do not expand these virtual children; retain the table and the rest of the UI,
|
||||
and report `managed_descendants_omitted` as a bounded capture. This limitation
|
||||
preserves identities for captured controls without claiming complete sheet
|
||||
content. Use screenshots for omitted content. No application-specific process
|
||||
restart or event-driven accessibility subsystem is introduced.
|
||||
|
||||
## Ordinary-layout collection
|
||||
|
||||
The ordinary Calc window also exposed a large hidden menu tree. An independent
|
||||
bounded census found 1,861 nodes, of which 1,638 were not showing, and 11,188
|
||||
explicit D-Bus calls without errors. The diagnostic final3 SDK completed 770
|
||||
nodes before its 25-second deadline; 752 were not showing. No individual call
|
||||
timed out. This was cumulative traversal work, not a stuck LibreOffice method.
|
||||
Native timed wrappers include peer-to-peer traffic and are not a count of bus
|
||||
messages; the independent census has a separate sequential-call timing scope.
|
||||
|
||||
Read role, state and supported interfaces together, then omit hidden native
|
||||
menu nodes and their subtrees before reading names, details or children. Keep
|
||||
showing, focused, selected or expanded menus, and do not apply the rule within
|
||||
web content or when visibility is unknown. A menu opened by the user is read
|
||||
on the next observation. Report `hidden_menu_subtrees_omitted` as an explicit
|
||||
bounded capture, preserving identities only for controls actually captured.
|
||||
Omitted commands consume no actionable index; tokens resolve by captured AT-SPI
|
||||
identity so opening a menu does not retarget an existing token by row number.
|
||||
|
||||
Collection now receives the snapshot's absolute deadline, including setup and
|
||||
the subsequent bounds phase. Keep accumulated nodes and bounds outside the
|
||||
cancelled futures so a slow later request cannot discard all prior work.
|
||||
Deadline exhaustion remains an incomplete read and cannot retain stable
|
||||
revision tokens. The native 25-second maximum is unchanged. The facade returns
|
||||
`walk_deadline_reached` and `element_bounds_timeout` captures without its usual
|
||||
automatic read retry; short read failures retain one retry. Real fault injection
|
||||
showed that retrying an exhausted capture otherwise takes about 51 seconds for
|
||||
two native reads. A later explicit observation can recover normally.
|
||||
|
||||
## Validation
|
||||
|
||||
Compare compact output against the same complete captured tree, checking
|
||||
retained content and actionable identities as well as row/character reduction.
|
||||
Exercise static-text changes, no-change revisions, field clearing, table rows,
|
||||
web boundaries, current tokens and deep layouts. Run real SDK observations and
|
||||
independent application readback on the owned Linux desktop.
|
||||
|
||||
Measure LibreOffice and the SDK process before, during and after single and
|
||||
repeated observations; retain raw samples and operation status. Compare the
|
||||
unchanged baseline and candidate against the same document. Do not infer
|
||||
provider token costs or historical OOM causality from shorter text or a timeout.
|
||||
|
||||
The memory-only candidate (`6babbd96`) passed four full-screen observations,
|
||||
but three ordinary-layout observations timed out at 25.762, 25.042 and 25.048
|
||||
seconds. This exact-binary comparison, followed by the instrumentation above,
|
||||
established the separate hidden-menu traversal problem.
|
||||
|
||||
The ordinary-layout repair (`d4ba57b6`) returned useful Task035 trees in 11.570,
|
||||
10.989, 13.454 and 10.815 seconds, with 157/157/190/157 actionable elements.
|
||||
All four reached the real table and retained its token; the unchanged second
|
||||
read returned `no_change`. File activation exposed 33 additional menu labels,
|
||||
and Escape restored the initial menu-label set. Both intentional omission
|
||||
reasons were reported, with `captureReadComplete=true` and `captureComplete=false`.
|
||||
Continuous samples over 68.986 seconds showed LibreOffice RSS between 340644
|
||||
and 340652 KiB, with a maximum sampling interval of 0.163 seconds. This is
|
||||
bounded real-application acceptance, not a long-duration leak or benchmark test.
|
||||
|
||||
The same native passed a real GTK comparison: legacy tree text stayed
|
||||
byte-identical, control depth fell from 10 to 2, static instructions survived,
|
||||
and token click, field replacement/readback and no-change/token stability all
|
||||
passed. Revision text grew from 352 to 554 characters because the baseline
|
||||
omitted passive information; this is not a measured token-cost reduction.
|
||||
|
||||
Controlled AT-SPI providers verified completed node and bounds preservation,
|
||||
continued traversal after a failed child, and no descendant enumeration after
|
||||
a failed state read. The old facade made two reads in 50.785 seconds for the
|
||||
metadata deadline and 50.965 seconds for the bounds deadline; these are
|
||||
before-fix retry measurements, not the intended latency of 0.20.9.
|
||||
|
||||
The final 0.20.9 package (`3554fe81` native, `f0ba1e38` facade) repeated the
|
||||
metadata deadline in 25.833 seconds with exactly one native snapshot, retaining
|
||||
26 controls and issuing no bounds requests after collection expired. A fresh
|
||||
ordinary maximized Calc run returned 157 controls in 11.084 seconds, then
|
||||
`no_change` in 10.492 seconds, with the same table token and both bounded-capture
|
||||
reasons. LibreOffice RSS stayed at 340228 KiB over 23.471 seconds of samples,
|
||||
with a maximum interval of 0.153 seconds. This final smoke used the packed
|
||||
0.20.9 SDK and verified native, JavaScript, package and canonical skill hashes.
|
||||
|
||||
Final version checks: Linux unit tests 294 passed/4 ignored; SDK tests 164
|
||||
passed/6 environment-dependent cases skipped; canonical skill 7 passed;
|
||||
release workflow tests 68 passed/1 skipped. The release build and package
|
||||
verifier passed pack, publish dry-run, checksum-based clean installation,
|
||||
canonical skill byte comparison and native loading (60 tools, revision support).
|
||||
Node REPL remains 0.1.5. No release was published by these checks.
|
||||
@@ -0,0 +1,112 @@
|
||||
# Linux 观察内存与紧凑化
|
||||
|
||||
[English](linux-observation-memory-and-compaction.md)
|
||||
|
||||
## 范围
|
||||
|
||||
在现有 Linux 第二轮候选上只补两项：可访问性树紧凑展示、LibreOffice 观察资源限制。
|
||||
App API、剪贴板和 OSWorld 代码不在范围内。补齐 SDK/driver 0.20.9 及其版本引用；
|
||||
正式发布另行执行。SDK 到原生 AT-SPI 的调用结构、权限和安装方式保持不变。
|
||||
|
||||
## 紧凑展示
|
||||
|
||||
此前候选只缩短了单行字段，没有迁移 macOS 0.20.6 已包含的结构投影。
|
||||
Linux 的 revision 还只包含可操作节点，导致静态状态和说明文本丢失。
|
||||
|
||||
保留观察到的静态上下文，再折叠空布局容器、去掉父子重复标签、合并同一父节点下
|
||||
相邻的静态文本。保留窗口、对话框、菜单、列表、表格和文档边界，区分网页正文与
|
||||
应用界面，保留焦点、选择、禁用状态及全部可操作节点。投影采用迭代实现，避免深树
|
||||
消耗调用栈。原生索引、身份和结构化动作列表不变。revision 比较包含保留下来的
|
||||
静态文字和可编辑字段清空后的空值。紧凑化仅用于模型观察输出；浏览器内部设置和
|
||||
授权提示的精确匹配仍使用原始静态标签边界。
|
||||
|
||||
## 内存调查
|
||||
|
||||
Task035 原始工作簿的表格暴露 1,073,741,824 个子项，并声明 `manages-descendants`。
|
||||
受控单次 `GetChildren` 请求使 LibreOffice RSS 在 24.14 秒内增加 107.75 MiB，
|
||||
拟合速率 4.45 MiB/s，请求进程内存不变。达到软限制后主动停止独立实例，无须真正
|
||||
触发 OOM 即可复现增长。这是直接 AT-SPI 请求的复现，不是历史沙箱重启根因的证明。
|
||||
|
||||
增加请求日志的原 b30 候选也通过实际 SDK 调用复现了增长：三次观察均退回 X11，
|
||||
日志停在表格 GetChildren 且未返回；第三次调用期间 LibreOffice 约增长 4.52 MiB/s，
|
||||
Node RSS 不变。调用方返回并不能取消应用已经开始的全量枚举。
|
||||
|
||||
普通节点改用 `ChildCount` 和惰性的 `GetChildAtIndex`，每层只保存下一兄弟游标；在请求
|
||||
子节点之前检查节点、深度和时间预算。快照与动作共用前序遍历及可操作索引计算。达到
|
||||
深度限制时只读元数据，不展开子项；子项请求失败时标记捕获不完整。修复不禁用
|
||||
Calc 可访问性，也不会先分配完整数组再截断。
|
||||
|
||||
实测证明仅按索引读取仍不够：对相同 512 个真实单元格连续读三轮，每轮约 2.8 秒
|
||||
返回，但 LibreOffice 每轮仍保留约 3.3–3.4 MiB 内存，部分单元格每次产生新的 AX
|
||||
对象路径。AT-SPI 明确说明不应枚举 `ManagesDescendants` 节点的子项。因此不展开
|
||||
这类虚拟子项，保留表格和其他界面，并以 `managed_descendants_omitted` 标记有界
|
||||
观察；已捕获控件仍保留稳定身份，但不声称包含完整工作表。缺失内容用截图查看。
|
||||
不引入应用重启或新的可访问性事件处理系统。
|
||||
|
||||
## 普通布局采集
|
||||
|
||||
普通 Calc 窗口还暴露了庞大的隐藏菜单树。独立有界普查读到 1,861 个节点，其中
|
||||
1,638 个未显示；11,188 次明确的 D-Bus 请求均无错误。带计时的 final3 SDK 在
|
||||
25 秒截止前完成 770 个节点，其中 752 个未显示，没有单次调用超时。因此剩余
|
||||
问题是大量正常遍历工作累积，不能归因为 LibreOffice 的某个方法卡死。
|
||||
原生计时包含点对点调用，计时包装次数不是总线消息数；独立普查的串行请求统计
|
||||
也不能直接当作原生并发遍历耗时。
|
||||
|
||||
并发读取角色、状态和接口后，在名称、详情和子项读取之前跳过隐藏原生菜单及其
|
||||
子树。保留已显示、聚焦、选中或展开的菜单；不对网页正文或未知可见性套用规则。
|
||||
打开菜单后的下一次观察会读取其内容。以 `hidden_menu_subtrees_omitted` 明确标记
|
||||
有界捕获，仅对实际读到的控件保留身份。
|
||||
省略的命令不占可操作索引；token 按捕获的 AT-SPI 身份解析，打开菜单不会因行号
|
||||
变化把已有 token 指向其他控件。
|
||||
|
||||
采集共享整个快照的绝对截止时间，包含准备和后续坐标阶段。累计节点和坐标保存在
|
||||
可取消 future 之外，后续慢请求超时不会抹掉先前结果。时间预算耗尽仍是读取不完整，
|
||||
不能保留稳定 revision token。原生 25 秒上限保持不变。facade 遇到
|
||||
`walk_deadline_reached` 或 `element_bounds_timeout` 时直接返回，不再自动重读；
|
||||
短暂读取失败仍保留一次重试。真实故障注入发现，对耗尽预算的捕获自动重试会产生
|
||||
两次原生读取、约 51 秒等待。后续显式观察仍可正常恢复。
|
||||
|
||||
## 验证
|
||||
|
||||
将紧凑输出与同一份完整捕获树比较，同时检查必要内容、可操作身份以及行数和字符
|
||||
变化。覆盖静态文本变化、无变化 revision、字段清空、表格行、网页边界、当前 token
|
||||
和深层布局。在独占的 Linux 测试桌面执行真实 SDK 观察和独立应用状态回读。
|
||||
|
||||
采集单次及重复观察之前、期间、之后的 LibreOffice 与 SDK 进程内存，保留原始采样
|
||||
及操作状态；基线和候选使用相同文档。不能从文本变短或调用超时推断模型计费下降
|
||||
或历史 OOM 根因。
|
||||
|
||||
只修复内存的候选（`6babbd96`）通过四次全屏观察，但三次普通布局观察分别在
|
||||
25.762、25.042、25.048 秒超时。这组精确二进制对照及上文计时验证，确认了
|
||||
独立的隐藏菜单遍历问题。
|
||||
|
||||
普通布局修复版（`d4ba57b6`）在同一 Task035 文档上四次分别耗时 11.570、10.989、
|
||||
13.454、10.815 秒，返回 157/157/190/157 个可操作元素。每次均到达真实表格，表格
|
||||
token 保持不变；第二次未变化读取返回 `no_change`。点击 File 后出现 33 个新增
|
||||
菜单标签，Escape 后恢复初始菜单标签集合。两项主动省略均明确报告，
|
||||
`captureReadComplete=true`、`captureComplete=false`。连续 68.986 秒采样中，
|
||||
LibreOffice RSS 在 340644–340652 KiB 之间，最大采样间隔 0.163 秒。
|
||||
这是有界的真实应用验收，不是长期泄漏压测或完整 benchmark。
|
||||
|
||||
同一 native 的真实 GTK 对照也通过：legacy 树逐字一致，控件深度从 10 降到 2，
|
||||
静态说明保留，token 点击、字段替换读回、无变化及 token 稳定性均正常。
|
||||
revision 字符从 352 增至 554，因为基线遗漏了静态信息；这不是 token 成本下降
|
||||
的测量结论。
|
||||
|
||||
受控 AT-SPI provider 验证了已完成节点和坐标保留、单个子项失败后继续读取后续
|
||||
兄弟节点、状态读取失败后不枚举子项。旧 facade 在元数据超时场景读取两次，耗时
|
||||
50.785 秒；坐标超时场景读取两次，耗时 50.965 秒。这些是修复前的重试测量，
|
||||
不是 0.20.9 的预期延迟。
|
||||
|
||||
最终 0.20.9 包（native `3554fe81`、facade `f0ba1e38`）在同一元数据超时场景
|
||||
耗时 25.833 秒，仅产生一次原生快照，保留 26 个控件，采集超时后没有发起坐标
|
||||
请求。新的普通最大化 Calc 验证在 11.084 秒返回 157 个控件，随后在 10.492 秒
|
||||
返回 `no_change`；表格 token 相同，两项有界捕获原因均明确报告。23.471 秒采样
|
||||
中 LibreOffice RSS 固定为 340228 KiB，最大采样间隔 0.153 秒。该最终 smoke
|
||||
使用打包后的 0.20.9 SDK，并核验了 native、JavaScript、包和固定 skill 的 hash。
|
||||
|
||||
最终版本检查：Linux 单元测试 294 通过、4 忽略；SDK 测试 164 通过、6 项依赖环境
|
||||
的测试跳过；固定 skill 7 通过；发布流程测试 68 通过、1 跳过。release build 和
|
||||
包验证通过，覆盖 pack、publish dry-run、checksum 校验的全新安装、固定 skill
|
||||
逐字比较和原生加载（60 个工具、支持 revision）。Node REPL 保持 0.1.5；这些
|
||||
检查没有执行正式发布。
|
||||
@@ -26,7 +26,7 @@ When first used, the skill runs these commands itself:
|
||||
|
||||
```bash
|
||||
qwen mcp add --scope user node-repl npx -y @qwen-code/[email protected]
|
||||
npm install --no-save --package-lock=false @qwen-code/[email protected].8
|
||||
npm install --no-save --package-lock=false @qwen-code/[email protected].9
|
||||
```
|
||||
|
||||
Restart Qwen Code after the MCP server is first added. The skill then resumes
|
||||
|
||||
@@ -47,7 +47,7 @@ If `node_repl` is unavailable, run:
|
||||
|
||||
```bash
|
||||
qwen mcp add --scope user node-repl npx -y @qwen-code/[email protected]
|
||||
npm install --no-save --package-lock=false @qwen-code/[email protected].8
|
||||
npm install --no-save --package-lock=false @qwen-code/[email protected].9
|
||||
```
|
||||
|
||||
Tell the user to restart Qwen Code, then stop. If only the SDK import is missing,
|
||||
@@ -245,6 +245,21 @@ captured element list even when `.text` reports a diff or no change. Use tokens
|
||||
from that list; unchanged tokens remain usable across observations. After a
|
||||
capture read failure, use only tokens issued by the latest observation.
|
||||
|
||||
Text rows omit default enabled/unselected states and the primary click action.
|
||||
`disabled` and `selected` mark non-default states; `actions` lists secondary
|
||||
actions. Editable content appears separately from its label as `value`, including
|
||||
an empty value after clearing a field. The element list retains all actions.
|
||||
Empty layout containers and duplicate labels are omitted; adjacent static text
|
||||
may share a row. Window, dialog, list and table context, focused/selected state,
|
||||
and actionable elements remain visible. Use element tokens, not row positions.
|
||||
Linux observations omit virtual children when an app manages their lifetime
|
||||
(`managed_descendants_omitted`). The returned tree is bounded; use a screenshot
|
||||
for sheet or canvas content that is not present in the tree.
|
||||
Linux also omits hidden native menu branches (`hidden_menu_subtrees_omitted`).
|
||||
Open a menu and observe again to read its displayed commands. A collection
|
||||
timeout returns the completed portion with an incomplete-capture indication;
|
||||
it does not establish that missing controls are absent.
|
||||
|
||||
Text is limited to 12,000 characters by default. Filter the elements for controls
|
||||
you need, or request `disableDiff: true` with a larger `maxTextChars` (minimum
|
||||
512). An omitted row does not prove absence. If you discarded earlier text,
|
||||
@@ -265,6 +280,10 @@ await computer.typeText({ ...target, text: 'hello' });
|
||||
nodeRepl.write((await computer.observeWindow(target)).text);
|
||||
```
|
||||
|
||||
When the controls and next actions are already known, combine the actions and
|
||||
saving in the same cell, then observe. A new observation is a decision boundary;
|
||||
do not split a known sequence into one call per action.
|
||||
|
||||
End the batch when opening a dialog or menu. For a dialog, list windows and
|
||||
observe the matching window before typing. For a menu in the same window,
|
||||
refresh its observation and use current menu tokens. Never guess a window ID.
|
||||
@@ -299,5 +318,7 @@ Do not stringify the whole observation or a raw driver result containing image
|
||||
bytes. In outer code mode, also forward each returned image block with `image()`
|
||||
as shown in the shared entrypoint, including images from `node_repl_wait`.
|
||||
|
||||
When all Computer Use work is complete, call `await computer.close()` and clear
|
||||
`globalThis.computer`. Reset the REPL only when no other persistent state is needed.
|
||||
Include `await computer.close()` and clearing `globalThis.computer` at the end
|
||||
of the cell that emits final verification. Inspect that result before reporting
|
||||
success; reconnect if it reveals unfinished work. Avoid a separate cleanup-only
|
||||
call. Reset the REPL only when no other persistent state is needed.
|
||||
@@ -14,18 +14,18 @@ documented in [`.vendored-patches.md`](.vendored-patches.md) and
|
||||
macOS and Linux:
|
||||
|
||||
```bash
|
||||
CUA_DRIVER_RS_VERSION=0.20.8 \
|
||||
CUA_DRIVER_RS_VERSION=0.20.9 \
|
||||
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/QwenLM/qwen-code/main/packages/cua-driver/scripts/install.sh)"
|
||||
```
|
||||
|
||||
Windows PowerShell:
|
||||
|
||||
```powershell
|
||||
$env:CUA_DRIVER_RS_VERSION = "0.20.8"
|
||||
$env:CUA_DRIVER_RS_VERSION = "0.20.9"
|
||||
irm https://raw.githubusercontent.com/QwenLM/qwen-code/main/packages/cua-driver/scripts/install.ps1 | iex
|
||||
```
|
||||
|
||||
Expected: qwen-cua-driver 0.20.8.
|
||||
Expected: qwen-cua-driver 0.20.9.
|
||||
|
||||
The released product uses Qwen-owned identities throughout:
|
||||
|
||||
|
||||
Generated
+13
-13
@@ -981,7 +981,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cua-driver"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -1017,7 +1017,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cua-driver-bindgen"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"cbindgen",
|
||||
"uniffi",
|
||||
@@ -1025,7 +1025,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cua-driver-contract"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"schemars",
|
||||
"serde",
|
||||
@@ -1035,7 +1035,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cua-driver-core"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -1064,7 +1064,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cua-driver-sdk"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"core-foundation",
|
||||
@@ -1086,7 +1086,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cua-driver-testkit"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"core-foundation",
|
||||
"libc",
|
||||
@@ -1100,7 +1100,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cua-driver-uia"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"cua-driver-core",
|
||||
@@ -1115,7 +1115,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cursor-overlay"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"cua-driver-contract",
|
||||
@@ -1132,7 +1132,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cursor-theme-cli"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"cursor-overlay",
|
||||
@@ -2900,7 +2900,7 @@ checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
|
||||
|
||||
[[package]]
|
||||
name = "pip-preview"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde_json",
|
||||
@@ -2960,7 +2960,7 @@ checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6"
|
||||
|
||||
[[package]]
|
||||
name = "platform-linux"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"ashpd",
|
||||
@@ -3006,7 +3006,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "platform-macos"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -3042,7 +3042,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "platform-windows"
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
|
||||
@@ -17,7 +17,7 @@ members = [
|
||||
]
|
||||
|
||||
[workspace.package]
|
||||
version = "0.20.8"
|
||||
version = "0.20.9"
|
||||
edition = "2021"
|
||||
authors = ["trycua"]
|
||||
license = "MIT"
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: cua-driver
|
||||
description: Drive a native GUI app (macOS, Windows, Linux) via the Qwen Cua Driver CLI (default) or MCP server; snapshot its accessibility tree, act through snapshot-bound element tokens, native menu paths, exact window geometry, or pixel coordinates, and verify from fresh state. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real application on the host.
|
||||
version: 0.20.8 # x-release-please-version
|
||||
version: 0.20.9 # x-release-please-version
|
||||
metadata:
|
||||
openclaw:
|
||||
requires:
|
||||
|
||||
@@ -1 +1 @@
|
||||
0.20.8
|
||||
0.20.9
|
||||
@@ -122,6 +122,7 @@ mod tests {
|
||||
checked: None,
|
||||
enabled: Some(true),
|
||||
selected: None,
|
||||
focused: None,
|
||||
description: None,
|
||||
depth: 0,
|
||||
parent_element_index: None,
|
||||
|
||||
@@ -13,6 +13,7 @@ use anyhow::Result;
|
||||
|
||||
pub mod cache;
|
||||
pub mod native;
|
||||
pub(crate) mod projection;
|
||||
pub mod revision;
|
||||
pub use cache::ElementCache;
|
||||
pub use native::ensure_listener_active;
|
||||
@@ -23,7 +24,7 @@ pub struct AtspiIdentity {
|
||||
pub object_path: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct AtspiNode {
|
||||
pub element_index: Option<usize>,
|
||||
pub role: String,
|
||||
@@ -35,6 +36,7 @@ pub struct AtspiNode {
|
||||
pub enabled: Option<bool>,
|
||||
/// Toggle/selection state for selectable controls.
|
||||
pub selected: Option<bool>,
|
||||
pub focused: Option<bool>,
|
||||
pub description: Option<String>,
|
||||
pub actions: Vec<String>,
|
||||
/// For AT-SPI: element_key = element_index as u64.
|
||||
@@ -87,7 +89,13 @@ impl AtspiTreeResult {
|
||||
|| (self.truncated
|
||||
&& !self.incomplete_notes.is_empty()
|
||||
&& self.incomplete_notes.iter().all(|note| {
|
||||
matches!(note.as_str(), "max_elements_reached" | "max_depth_reached")
|
||||
matches!(
|
||||
note.as_str(),
|
||||
"max_elements_reached"
|
||||
| "max_depth_reached"
|
||||
| "managed_descendants_omitted"
|
||||
| "hidden_menu_subtrees_omitted"
|
||||
)
|
||||
})))
|
||||
}
|
||||
}
|
||||
@@ -140,50 +148,31 @@ pub fn walk_tree_bounded(
|
||||
max_elements: Option<usize>,
|
||||
max_depth: Option<usize>,
|
||||
) -> AtspiTreeResult {
|
||||
// Native AT-SPI (most complete). On a COLD launch the Qt6 (and some GTK)
|
||||
// AT-SPI bridge registers lazily — the first walk against a freshly
|
||||
// launched app can come back with just the root window (element_count=1,
|
||||
// no children) because `org.a11y.atspi.Registry` hasn't finished
|
||||
// enumerating the app's tree yet. Retry a few times with a short backoff
|
||||
// while the tree is suspiciously root-only, so the first get_window_state
|
||||
// after launch returns the real tree instead of an empty one. See #1927.
|
||||
const MAX_ATTEMPTS: usize = 4;
|
||||
let mut native_failure = None;
|
||||
for attempt in 0..MAX_ATTEMPTS {
|
||||
match native::walk_tree_bounded(pid, xid, max_elements, max_depth) {
|
||||
Ok(Some(walked)) => {
|
||||
// `nodes.len() <= 1` == only the root window resolved: the
|
||||
// cold-registry symptom. Accept any real tree immediately; only
|
||||
// keep waiting on the degenerate case, and accept it anyway on the
|
||||
// final attempt rather than discarding a (minimal) valid result.
|
||||
if !walked.markdown.is_empty()
|
||||
&& (walked.nodes.len() > 1 || attempt == MAX_ATTEMPTS - 1)
|
||||
{
|
||||
let md = if let Some(q) = query {
|
||||
filter_tree(&walked.markdown, q)
|
||||
} else {
|
||||
walked.markdown
|
||||
};
|
||||
return AtspiTreeResult {
|
||||
tree_markdown: md,
|
||||
nodes: walked.nodes,
|
||||
bounds: walked.bounds,
|
||||
trusted: true,
|
||||
degraded_reason: None,
|
||||
window_scoped: walked.window_scoped,
|
||||
backend: AtspiBackend::Atspi,
|
||||
complete: walked.complete && walked.window_scoped,
|
||||
truncated: walked.truncated,
|
||||
incomplete_notes: walked.incomplete_notes,
|
||||
};
|
||||
}
|
||||
}
|
||||
Ok(None) => {}
|
||||
Err(error) => native_failure = Some(error.to_string()),
|
||||
}
|
||||
if attempt < MAX_ATTEMPTS - 1 {
|
||||
std::thread::sleep(std::time::Duration::from_millis(150));
|
||||
match walk_with_cold_start_retries(native::OP_TIMEOUT, |remaining| {
|
||||
native::walk_tree_bounded_with_timeout(pid, xid, max_elements, max_depth, remaining)
|
||||
}) {
|
||||
Ok(Some(walked)) if !walked.markdown.is_empty() => {
|
||||
let md = if let Some(q) = query {
|
||||
filter_tree(&walked.markdown, q)
|
||||
} else {
|
||||
walked.markdown
|
||||
};
|
||||
return AtspiTreeResult {
|
||||
tree_markdown: md,
|
||||
nodes: walked.nodes,
|
||||
bounds: walked.bounds,
|
||||
trusted: true,
|
||||
degraded_reason: None,
|
||||
window_scoped: walked.window_scoped,
|
||||
backend: AtspiBackend::Atspi,
|
||||
complete: walked.complete && walked.window_scoped,
|
||||
truncated: walked.truncated,
|
||||
incomplete_notes: walked.incomplete_notes,
|
||||
};
|
||||
}
|
||||
Ok(_) => {}
|
||||
Err(error) => native_failure = Some(error.to_string()),
|
||||
}
|
||||
|
||||
// Fallback: X11 window properties as minimal tree.
|
||||
@@ -192,6 +181,35 @@ pub fn walk_tree_bounded(
|
||||
fallback
|
||||
}
|
||||
|
||||
fn walk_with_cold_start_retries(
|
||||
timeout: std::time::Duration,
|
||||
mut walk: impl FnMut(std::time::Duration) -> Result<Option<native::WalkedTree>>,
|
||||
) -> Result<Option<native::WalkedTree>> {
|
||||
const MAX_ATTEMPTS: usize = 4;
|
||||
const BACKOFF: std::time::Duration = std::time::Duration::from_millis(150);
|
||||
let deadline = std::time::Instant::now() + timeout;
|
||||
for attempt in 0..MAX_ATTEMPTS {
|
||||
let result = walk(deadline.saturating_duration_since(std::time::Instant::now()))?;
|
||||
// Qt/GTK can register a complete but root-only tree on cold launch.
|
||||
// Retry only that lazy-registration case, sharing the snapshot budget.
|
||||
// Incomplete reads must reach the caller, not multiply its own retries.
|
||||
let retry = result.as_ref().is_some_and(|walked| {
|
||||
walked.complete && (walked.nodes.len() <= 1 || walked.markdown.is_empty())
|
||||
});
|
||||
if !retry
|
||||
|| attempt == MAX_ATTEMPTS - 1
|
||||
|| deadline.saturating_duration_since(std::time::Instant::now()) <= BACKOFF
|
||||
{
|
||||
return Ok(result);
|
||||
}
|
||||
std::thread::sleep(BACKOFF);
|
||||
if std::time::Instant::now() >= deadline {
|
||||
return Ok(result);
|
||||
}
|
||||
}
|
||||
unreachable!()
|
||||
}
|
||||
|
||||
/// Perform the first advertised action on element `idx` within pid's app tree.
|
||||
/// Returns `Ok((action_name, suspected_noop))` on success — `suspected_noop`
|
||||
/// is true when the actuated node looked like a silent no-op (a passive
|
||||
@@ -383,6 +401,7 @@ fn walk_via_x11_properties(xid: u64, query: Option<&str>) -> AtspiTreeResult {
|
||||
checked: None,
|
||||
enabled: None,
|
||||
selected: None,
|
||||
focused: None,
|
||||
description: if wm_class.is_empty() {
|
||||
None
|
||||
} else {
|
||||
@@ -436,22 +455,33 @@ pub(crate) fn format_revision_body(node: &AtspiNode) -> String {
|
||||
format!("<{}>", node.role),
|
||||
serde_json::to_string(label).expect("string labels serialize"),
|
||||
];
|
||||
if let Some(value) = node.value.as_deref().filter(|value| !value.is_empty()) {
|
||||
if let Some(value) = node.value.as_deref().filter(|value| *value != label) {
|
||||
fields.push(format!(
|
||||
"value={}",
|
||||
serde_json::to_string(value).expect("string values serialize")
|
||||
));
|
||||
}
|
||||
if let Some(enabled) = node.enabled {
|
||||
fields.push(format!("enabled={enabled}"));
|
||||
if node.enabled == Some(false) {
|
||||
fields.push("disabled".into());
|
||||
}
|
||||
if let Some(selected) = node.selected.or(node.checked) {
|
||||
fields.push(format!("selected={selected}"));
|
||||
if node.focused == Some(true) {
|
||||
fields.push("focused".into());
|
||||
}
|
||||
if !node.actions.is_empty() {
|
||||
if node.selected.or(node.checked) == Some(true) {
|
||||
fields.push("selected".into());
|
||||
}
|
||||
let primary = native::activation_index(&node.role, &node.actions);
|
||||
let secondary = node
|
||||
.actions
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(index, action)| Some(*index) != primary && !action.is_empty())
|
||||
.map(|(_, action)| action)
|
||||
.collect::<Vec<_>>();
|
||||
if !secondary.is_empty() {
|
||||
fields.push(format!(
|
||||
"actions={}",
|
||||
serde_json::to_string(&node.actions).expect("string actions serialize")
|
||||
serde_json::to_string(&secondary).expect("string actions serialize")
|
||||
));
|
||||
}
|
||||
if node.in_web_content {
|
||||
@@ -543,3 +573,126 @@ fn filter_tree(markdown: &str, query: &str) -> String {
|
||||
r.push('\n');
|
||||
r
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod retry_tests {
|
||||
use super::*;
|
||||
use std::time::Duration;
|
||||
|
||||
fn tree(count: usize, complete: bool) -> native::WalkedTree {
|
||||
native::WalkedTree {
|
||||
markdown: "root".into(),
|
||||
nodes: (0..count)
|
||||
.map(|index| AtspiNode {
|
||||
element_index: Some(index),
|
||||
role: "frame".into(),
|
||||
name: None,
|
||||
value: None,
|
||||
checked: None,
|
||||
enabled: None,
|
||||
selected: None,
|
||||
focused: None,
|
||||
description: None,
|
||||
actions: Vec::new(),
|
||||
element_key: index as u64,
|
||||
depth: 0,
|
||||
parent_element_index: None,
|
||||
in_web_content: false,
|
||||
identity: None,
|
||||
})
|
||||
.collect(),
|
||||
bounds: Vec::new(),
|
||||
window_scoped: true,
|
||||
complete,
|
||||
truncated: false,
|
||||
incomplete_notes: if complete {
|
||||
Vec::new()
|
||||
} else {
|
||||
vec!["interfaces_unavailable".into()]
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn complete_trees_and_incomplete_reads_return_immediately() {
|
||||
for (count, complete) in [(2, true), (0, false), (1, false), (2, false)] {
|
||||
let mut calls = 0;
|
||||
let result = walk_with_cold_start_retries(native::OP_TIMEOUT, |_| {
|
||||
calls += 1;
|
||||
Ok(Some(tree(count, complete)))
|
||||
})
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(calls, 1);
|
||||
assert_eq!(result.complete, complete);
|
||||
assert_eq!(result.nodes.len(), count);
|
||||
assert_eq!(
|
||||
result.incomplete_notes,
|
||||
tree(count, complete).incomplete_notes
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unavailable_and_failed_reads_return_immediately() {
|
||||
let mut calls = 0;
|
||||
assert!(walk_with_cold_start_retries(native::OP_TIMEOUT, |_| {
|
||||
calls += 1;
|
||||
Ok(None)
|
||||
})
|
||||
.unwrap()
|
||||
.is_none());
|
||||
assert_eq!(calls, 1);
|
||||
calls = 0;
|
||||
let error = walk_with_cold_start_retries(native::OP_TIMEOUT, |_| {
|
||||
calls += 1;
|
||||
Err(anyhow::anyhow!("transport unavailable"))
|
||||
})
|
||||
.err()
|
||||
.unwrap();
|
||||
assert_eq!(calls, 1);
|
||||
assert_eq!(error.to_string(), "transport unavailable");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cold_root_can_recover_within_the_same_budget() {
|
||||
let mut budgets = Vec::new();
|
||||
let result = walk_with_cold_start_retries(native::OP_TIMEOUT, |budget| {
|
||||
budgets.push(budget);
|
||||
Ok(Some(tree(budgets.len(), true)))
|
||||
})
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(budgets.len(), 2);
|
||||
assert!(budgets[0] - budgets[1] >= Duration::from_millis(150));
|
||||
assert_eq!(result.nodes.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cold_root_is_preserved_after_last_attempt() {
|
||||
let mut calls = 0;
|
||||
let result = walk_with_cold_start_retries(native::OP_TIMEOUT, |_| {
|
||||
calls += 1;
|
||||
Ok(Some(tree(1, true)))
|
||||
})
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(calls, 4);
|
||||
assert_eq!(result.markdown, "root");
|
||||
assert!(result.complete);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn exhausted_budget_preserves_root_without_another_attempt() {
|
||||
let mut calls = 0;
|
||||
let result = walk_with_cold_start_retries(Duration::ZERO, |budget| {
|
||||
calls += 1;
|
||||
assert!(budget.is_zero());
|
||||
Ok(Some(tree(1, true)))
|
||||
})
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(calls, 1);
|
||||
assert_eq!(result.markdown, "root");
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,268 @@
|
||||
use super::AtspiNode;
|
||||
|
||||
fn label(node: &AtspiNode) -> &str {
|
||||
node.name
|
||||
.as_deref()
|
||||
.or(node.value.as_deref())
|
||||
.or(node.description.as_deref())
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
fn protected(node: &AtspiNode) -> bool {
|
||||
node.element_index.is_some()
|
||||
|| node.focused == Some(true)
|
||||
|| node.selected == Some(true)
|
||||
|| node.checked == Some(true)
|
||||
|| !node.actions.is_empty()
|
||||
}
|
||||
|
||||
fn label_only(node: &AtspiNode) -> bool {
|
||||
node.enabled != Some(false)
|
||||
&& [
|
||||
node.name.as_deref(),
|
||||
node.value.as_deref(),
|
||||
node.description.as_deref(),
|
||||
]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.all(|value| value == label(node))
|
||||
}
|
||||
|
||||
fn text_only(node: &AtspiNode, leaf: bool) -> bool {
|
||||
leaf && !protected(node)
|
||||
&& label_only(node)
|
||||
&& matches!(node.role.as_str(), "label" | "static" | "static text")
|
||||
}
|
||||
|
||||
fn context(node: &AtspiNode) -> bool {
|
||||
matches!(
|
||||
node.role.as_str(),
|
||||
"frame"
|
||||
| "window"
|
||||
| "dialog"
|
||||
| "alert"
|
||||
| "menu"
|
||||
| "menu bar"
|
||||
| "table"
|
||||
| "table row"
|
||||
| "table cell"
|
||||
| "list"
|
||||
| "list item"
|
||||
| "tree"
|
||||
| "tree item"
|
||||
| "page tab list"
|
||||
) || node.role.contains("document")
|
||||
}
|
||||
|
||||
/// Project presentation only: native actionable indices and identities never change.
|
||||
pub(crate) fn compact(nodes: Vec<AtspiNode>) -> Vec<AtspiNode> {
|
||||
let mut ends = vec![nodes.len(); nodes.len()];
|
||||
let mut ancestry: Vec<usize> = Vec::new();
|
||||
for (index, node) in nodes.iter().enumerate() {
|
||||
while ancestry
|
||||
.last()
|
||||
.is_some_and(|&parent| nodes[parent].depth >= node.depth)
|
||||
{
|
||||
ends[ancestry.pop().expect("checked ancestor")] = index;
|
||||
}
|
||||
ancestry.push(index);
|
||||
}
|
||||
let mut skip = vec![false; nodes.len()];
|
||||
for (index, node) in nodes.iter().enumerate() {
|
||||
if protected(node) || context(node) || !label_only(node) {
|
||||
continue;
|
||||
}
|
||||
let leaf = ends[index] == index + 1;
|
||||
let layout = matches!(
|
||||
node.role.as_str(),
|
||||
"filler" | "panel" | "section" | "scroll pane" | "unknown"
|
||||
);
|
||||
skip[index] = (label(node).is_empty() && (layout || leaf))
|
||||
|| (layout
|
||||
&& !leaf
|
||||
&& ends[index + 1] == ends[index]
|
||||
&& nodes[index + 1].in_web_content == node.in_web_content
|
||||
&& label(&nodes[index + 1]) == label(node));
|
||||
}
|
||||
|
||||
let mut result: Vec<AtspiNode> = Vec::new();
|
||||
let mut parents: Vec<(usize, usize)> = Vec::new();
|
||||
let mut previous_text_parent = None;
|
||||
let mut previous_was_text = false;
|
||||
for (index, mut node) in nodes.into_iter().enumerate() {
|
||||
while parents
|
||||
.last()
|
||||
.is_some_and(|(depth, _)| *depth >= node.depth)
|
||||
{
|
||||
parents.pop();
|
||||
}
|
||||
if skip[index] {
|
||||
continue;
|
||||
}
|
||||
let source_depth = node.depth;
|
||||
let parent = parents.last().map(|&(_, index)| index);
|
||||
let text = text_only(&node, ends[index] == index + 1);
|
||||
if text
|
||||
&& parent.is_some_and(|parent| {
|
||||
let parent = &result[parent];
|
||||
parent.in_web_content == node.in_web_content
|
||||
&& !label(parent).is_empty()
|
||||
&& label(parent) == label(&node)
|
||||
})
|
||||
{
|
||||
continue;
|
||||
}
|
||||
if text && previous_was_text && previous_text_parent == parent {
|
||||
if let Some(previous) = result
|
||||
.last_mut()
|
||||
.filter(|previous| previous.in_web_content == node.in_web_content)
|
||||
{
|
||||
let combined = format!("{}\n{}", label(previous), label(&node));
|
||||
previous.name = Some(combined);
|
||||
previous.value = None;
|
||||
previous.description = None;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
node.depth = parents.len();
|
||||
node.parent_element_index = parent.and_then(|parent| {
|
||||
result[parent]
|
||||
.element_index
|
||||
.or(result[parent].parent_element_index)
|
||||
});
|
||||
previous_was_text = text;
|
||||
previous_text_parent = parent;
|
||||
parents.push((source_depth, result.len()));
|
||||
result.push(node);
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::atspi::AtspiIdentity;
|
||||
|
||||
fn node(role: &str, text: &str, depth: usize, index: Option<usize>) -> AtspiNode {
|
||||
AtspiNode {
|
||||
role: role.into(),
|
||||
name: Some(text.into()),
|
||||
depth,
|
||||
element_index: index,
|
||||
identity: Some(AtspiIdentity {
|
||||
unique_owner: ":1.2".into(),
|
||||
object_path: format!("/{role}/{depth}/{index:?}"),
|
||||
}),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn folds_layout_and_duplicate_labels_without_changing_controls() {
|
||||
let button = node("push button", "Save", 3, Some(7));
|
||||
let projected = compact(vec![
|
||||
node("frame", "Document", 0, None),
|
||||
node("panel", "", 1, None),
|
||||
node("panel", "Save", 2, None),
|
||||
button.clone(),
|
||||
node("label", "Save", 4, None),
|
||||
]);
|
||||
assert_eq!(projected.len(), 2);
|
||||
assert_eq!(projected[1].element_index, Some(7));
|
||||
assert_eq!(projected[1].identity, button.identity);
|
||||
assert_eq!(projected[1].depth, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merges_static_siblings_but_keeps_table_and_web_boundaries() {
|
||||
let mut web = node("label", "web", 1, None);
|
||||
web.in_web_content = true;
|
||||
let projected = compact(vec![
|
||||
node("frame", "Document", 0, None),
|
||||
node("label", "first", 1, None),
|
||||
node("static", "second", 1, None),
|
||||
web,
|
||||
node("table row", "", 1, None),
|
||||
node("label", "row one", 2, None),
|
||||
node("table row", "", 1, None),
|
||||
node("label", "row two", 2, None),
|
||||
]);
|
||||
assert_eq!(projected.len(), 7);
|
||||
assert_eq!(label(&projected[1]), "first\nsecond");
|
||||
assert_eq!(label(&projected[2]), "web");
|
||||
assert_eq!(label(&projected[4]), "row one");
|
||||
assert_eq!(label(&projected[6]), "row two");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn preserves_empty_values_states_actions_and_action_ancestry() {
|
||||
let mut field = node("entry", "Name", 0, Some(4));
|
||||
field.value = Some(String::new());
|
||||
let mut focused = node("panel", "", 1, None);
|
||||
focused.focused = Some(true);
|
||||
let mut disabled = node("push button", "", 1, None);
|
||||
disabled.enabled = Some(false);
|
||||
disabled.actions = vec!["click".into()];
|
||||
let mut selected = node("list item", "", 1, None);
|
||||
selected.selected = Some(true);
|
||||
let projected = compact(vec![
|
||||
field,
|
||||
focused,
|
||||
disabled,
|
||||
selected,
|
||||
node("entry", "Child", 1, Some(9)),
|
||||
]);
|
||||
assert_eq!(projected.len(), 5);
|
||||
assert_eq!(projected[0].value.as_deref(), Some(""));
|
||||
assert_eq!(projected[1].focused, Some(true));
|
||||
assert_eq!(projected[2].actions, ["click"]);
|
||||
assert_eq!(projected[3].selected, Some(true));
|
||||
assert_eq!(projected[4].parent_element_index, Some(4));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deep_empty_layout_is_iterative() {
|
||||
let mut nodes = (0..5000)
|
||||
.map(|depth| node("panel", "", depth, None))
|
||||
.collect::<Vec<_>>();
|
||||
nodes.push(node("entry", "End", 5000, Some(0)));
|
||||
let projected = compact(nodes);
|
||||
assert_eq!(projected.len(), 1);
|
||||
assert_eq!(projected[0].depth, 0);
|
||||
assert_eq!(projected[0].element_index, Some(0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redundant_form_shrinks_without_losing_field_values_or_tokens() {
|
||||
let mut nodes = vec![node("frame", "Profile", 0, None)];
|
||||
for index in 0..8 {
|
||||
nodes.push(node("panel", "", 1, None));
|
||||
let label = format!("Field {index}");
|
||||
let mut field = node("entry", &label, 2, Some(index));
|
||||
field.value = Some(format!("value {index}"));
|
||||
nodes.push(field);
|
||||
nodes.push(node("label", &label, 3, None));
|
||||
}
|
||||
let render = |nodes: &[AtspiNode]| {
|
||||
nodes
|
||||
.iter()
|
||||
.map(super::super::format_revision_body)
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n")
|
||||
};
|
||||
let full = render(&nodes);
|
||||
let projected = compact(nodes.clone());
|
||||
let compact_text = render(&projected);
|
||||
assert!(compact_text.len() < full.len());
|
||||
assert_eq!(projected.len(), 9);
|
||||
for original in nodes.iter().filter(|node| node.element_index.is_some()) {
|
||||
let retained = projected
|
||||
.iter()
|
||||
.find(|node| node.element_index == original.element_index)
|
||||
.unwrap();
|
||||
assert_eq!(retained.identity, original.identity);
|
||||
assert_eq!(retained.value, original.value);
|
||||
assert_eq!(retained.actions, original.actions);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -271,6 +271,7 @@ mod tests {
|
||||
checked: None,
|
||||
enabled: Some(true),
|
||||
selected: None,
|
||||
focused: None,
|
||||
description: None,
|
||||
actions: vec!["click".into()],
|
||||
element_key: 0,
|
||||
@@ -301,7 +302,12 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn budget_captures_retain_lineage_but_failures_and_coverage_changes_invalidate_it() {
|
||||
for reason in ["max_elements_reached", "max_depth_reached"] {
|
||||
for reason in [
|
||||
"max_elements_reached",
|
||||
"max_depth_reached",
|
||||
"managed_descendants_omitted",
|
||||
"hidden_menu_subtrees_omitted",
|
||||
] {
|
||||
let revisions = LinuxObservationRevisions::new();
|
||||
let mut bounded = tree(vec![node("/button", "Before")]);
|
||||
bounded.complete = false;
|
||||
@@ -417,6 +423,139 @@ mod tests {
|
||||
assert_eq!(changed.nodes[0].element_id, initial.nodes[0].element_id);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn named_editable_changes_and_clearing_remain_visible() {
|
||||
let revisions = LinuxObservationRevisions::new();
|
||||
let mut captured = tree(
|
||||
(0..8)
|
||||
.map(|index| {
|
||||
let mut item = node(&format!("/field/{index}"), "Name");
|
||||
item.element_index = Some(index);
|
||||
item.role = "entry".into();
|
||||
item.value = Some(String::new());
|
||||
item
|
||||
})
|
||||
.collect(),
|
||||
);
|
||||
let mut base = None;
|
||||
let mut element_id = None;
|
||||
for (value, mode) in [
|
||||
("", ObservationMode::Full),
|
||||
("prefix|X", ObservationMode::Diff),
|
||||
("prefix|X", ObservationMode::NoChange),
|
||||
("", ObservationMode::Diff),
|
||||
] {
|
||||
captured.nodes[0].value = Some(value.into());
|
||||
let observation = revisions
|
||||
.observe(
|
||||
session(),
|
||||
10,
|
||||
20,
|
||||
5000,
|
||||
usize::MAX,
|
||||
&captured,
|
||||
&request(base),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(observation.mode, mode);
|
||||
assert!(observation.nodes[0]
|
||||
.body
|
||||
.contains(&format!("value={value:?}")));
|
||||
if let Some(id) = &element_id {
|
||||
assert_eq!(&observation.nodes[0].element_id, id);
|
||||
}
|
||||
element_id = Some(observation.nodes[0].element_id.clone());
|
||||
base = Some(observation.revision_id);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compact_rows_preserve_text_state_and_secondary_actions() {
|
||||
let mut control = node("/entry", "Line 1\n\"Line 2\"");
|
||||
control.role = "entry".into();
|
||||
control.value = control.name.clone();
|
||||
control.enabled = Some(false);
|
||||
control.selected = Some(true);
|
||||
control.actions = vec!["activate".into(), "showContextMenu".into()];
|
||||
let body = format_revision_body(&control);
|
||||
assert_eq!(
|
||||
body,
|
||||
"<entry> \"Line 1\\n\\\"Line 2\\\"\" disabled selected actions=[\"showContextMenu\"]"
|
||||
);
|
||||
let disabled = body.clone();
|
||||
control.enabled = Some(true);
|
||||
assert_ne!(format_revision_body(&control), disabled);
|
||||
let selected = format_revision_body(&control);
|
||||
control.selected = Some(false);
|
||||
assert_ne!(format_revision_body(&control), selected);
|
||||
let secondary = format_revision_body(&control);
|
||||
control.actions.pop();
|
||||
assert_ne!(format_revision_body(&control), secondary);
|
||||
assert_eq!(control.actions, ["activate"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compact_static_text_changes_are_visible_without_remapping_controls() {
|
||||
let revisions = LinuxObservationRevisions::new();
|
||||
let capture = |message: &str| {
|
||||
let mut root = node("/window", "Dialog");
|
||||
root.role = "dialog".into();
|
||||
root.element_index = None;
|
||||
root.actions.clear();
|
||||
let mut text = node("/message", message);
|
||||
text.role = "label".into();
|
||||
text.element_index = None;
|
||||
text.actions.clear();
|
||||
text.depth = 1;
|
||||
let mut button = node("/ok", "OK");
|
||||
button.element_index = Some(17);
|
||||
button.depth = 1;
|
||||
tree(super::super::projection::compact(vec![root, text, button]))
|
||||
};
|
||||
let first = revisions
|
||||
.observe(session(), 10, 20, 50, 5, &capture("Saved"), &request(None))
|
||||
.unwrap();
|
||||
let unchanged = revisions
|
||||
.observe(
|
||||
session(),
|
||||
10,
|
||||
20,
|
||||
50,
|
||||
5,
|
||||
&capture("Saved"),
|
||||
&request(Some(first.revision_id.clone())),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(unchanged.mode, ObservationMode::NoChange);
|
||||
let changed = revisions
|
||||
.observe(
|
||||
session(),
|
||||
10,
|
||||
20,
|
||||
50,
|
||||
5,
|
||||
&capture("Failed to save"),
|
||||
&request(Some(unchanged.revision_id)),
|
||||
)
|
||||
.unwrap();
|
||||
assert!(changed.text.contains("Failed to save"));
|
||||
let control =
|
||||
|result: &cua_driver_core::observation_revision::ObservationRevisionResult| {
|
||||
result
|
||||
.nodes
|
||||
.iter()
|
||||
.find(|node| node.actionable_index == Some(17))
|
||||
.unwrap()
|
||||
.element_id
|
||||
};
|
||||
assert_eq!(control(&first), control(&changed));
|
||||
assert!(changed
|
||||
.nodes
|
||||
.iter()
|
||||
.filter(|node| node.actionable_index.is_some())
|
||||
.all(|node| node.actionable_index == Some(17)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn owner_change_and_incomplete_capture_fail_closed() {
|
||||
let revisions = LinuxObservationRevisions::new();
|
||||
|
||||
@@ -319,6 +319,7 @@ mod tests {
|
||||
checked: None,
|
||||
enabled: None,
|
||||
selected: None,
|
||||
focused: None,
|
||||
description: None,
|
||||
actions: actions.iter().map(|value| (*value).to_owned()).collect(),
|
||||
element_key: 0,
|
||||
|
||||
@@ -662,6 +662,7 @@ mod tests {
|
||||
checked: None,
|
||||
enabled: None,
|
||||
selected: None,
|
||||
focused: None,
|
||||
description: None,
|
||||
actions: actions.iter().map(|value| (*value).to_owned()).collect(),
|
||||
element_key: 0,
|
||||
|
||||
@@ -1010,7 +1010,7 @@ pub fn with_x11_foreground<T>(
|
||||
unsafe {
|
||||
x11::xlib::XGetInputFocus(display, &mut prior_core_focus, &mut prior_revert);
|
||||
}
|
||||
let timeout = std::time::Duration::from_millis(settle_ms.max(400));
|
||||
let timeout = std::time::Duration::from_millis(settle_ms.max(1000));
|
||||
let started = std::time::Instant::now();
|
||||
let deadline = started + timeout;
|
||||
let retry_at = started + timeout / 2;
|
||||
|
||||
@@ -841,13 +841,19 @@ impl Tool for GetWindowStateTool {
|
||||
let query_for_walk = query.clone();
|
||||
|
||||
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
|
||||
let tree_result = Some(crate::atspi::walk_tree_bounded(
|
||||
let mut tree = crate::atspi::walk_tree_bounded(
|
||||
pid,
|
||||
xid,
|
||||
query_for_walk.as_deref(),
|
||||
max_elements,
|
||||
max_depth,
|
||||
));
|
||||
);
|
||||
// Compact model observations only. Internal browser setup/consent
|
||||
// matching needs the original label boundaries from the native tree.
|
||||
if revision_request_for_capture.is_some() {
|
||||
tree.nodes = crate::atspi::projection::compact(tree.nodes);
|
||||
}
|
||||
let tree_result = Some(tree);
|
||||
// Bounds and element indices come from the same captured AT-SPI
|
||||
// traversal. Joining two live walks by ordinal mis-associated
|
||||
// Chromium controls when its lazy subtree changed between walks.
|
||||
@@ -1051,7 +1057,7 @@ impl Tool for GetWindowStateTool {
|
||||
// the text from a caller reading the structured side,
|
||||
// leaving it only in tree_markdown. See the macOS
|
||||
// get_window_state builder for the rationale.
|
||||
if let Some(value) = n.value.clone().filter(|v| !v.is_empty()) {
|
||||
if let Some(value) = n.value.clone() {
|
||||
entry["value"] = json!(value);
|
||||
}
|
||||
if let Some(enabled) = n.enabled {
|
||||
@@ -1060,6 +1066,9 @@ impl Tool for GetWindowStateTool {
|
||||
if !n.actions.is_empty() {
|
||||
entry["actions"] = json!(n.actions);
|
||||
}
|
||||
if let Some(focused) = n.focused {
|
||||
entry["focused"] = json!(focused);
|
||||
}
|
||||
if let Some(selected) = n.selected {
|
||||
entry["selected"] = json!(selected);
|
||||
}
|
||||
@@ -1653,7 +1662,7 @@ fn chromium_family_program(program: &str) -> bool {
|
||||
|
||||
// ── shared helpers ────────────────────────────────────────────────────────────
|
||||
|
||||
/// Resolve an AT-SPI element's center in window-local coordinates.
|
||||
/// Resolve a visible AT-SPI element point in window-local coordinates.
|
||||
///
|
||||
/// Returns `(xid, window_local_x, window_local_y)`.
|
||||
/// Looks up element bounds via native AT-SPI, finds the owning window
|
||||
@@ -1665,10 +1674,7 @@ fn resolve_element_local_coords(
|
||||
idx: usize,
|
||||
xid_hint: Option<u64>,
|
||||
) -> anyhow::Result<(u64, f64, f64)> {
|
||||
let (bx, by, bw, bh) = crate::atspi::get_element_bounds(pid, idx)?;
|
||||
let screen_cx = bx as f64 + bw as f64 / 2.0;
|
||||
let screen_cy = by as f64 + bh as f64 / 2.0;
|
||||
screen_point_to_window_local(pid, xid_hint, screen_cx, screen_cy)
|
||||
resolve_element_local_coords_exact(pid, idx, xid_hint, None)
|
||||
}
|
||||
|
||||
fn resolve_element_local_coords_exact(
|
||||
@@ -1676,21 +1682,6 @@ fn resolve_element_local_coords_exact(
|
||||
idx: usize,
|
||||
xid_hint: Option<u64>,
|
||||
identity: Option<crate::atspi::AtspiIdentity>,
|
||||
) -> anyhow::Result<(u64, f64, f64)> {
|
||||
let (bx, by, bw, bh) = match identity {
|
||||
Some(identity) => crate::atspi::get_element_bounds_exact(pid, idx, identity)?,
|
||||
None => crate::atspi::get_element_bounds(pid, idx)?,
|
||||
};
|
||||
let screen_cx = bx as f64 + bw as f64 / 2.0;
|
||||
let screen_cy = by as f64 + bh as f64 / 2.0;
|
||||
screen_point_to_window_local(pid, xid_hint, screen_cx, screen_cy)
|
||||
}
|
||||
|
||||
fn screen_point_to_window_local(
|
||||
pid: u32,
|
||||
xid_hint: Option<u64>,
|
||||
screen_cx: f64,
|
||||
screen_cy: f64,
|
||||
) -> anyhow::Result<(u64, f64, f64)> {
|
||||
let xid = if let Some(x) = xid_hint {
|
||||
x
|
||||
@@ -1707,14 +1698,14 @@ fn screen_point_to_window_local(
|
||||
.map(|w| w.xid)
|
||||
.ok_or_else(|| anyhow::anyhow!("No windows for pid {pid}"))?
|
||||
};
|
||||
let bounds = crate::atspi::native::get_element_bounds_in_window(pid, idx, xid, identity)?;
|
||||
|
||||
if crate::wayland::wayland_input_enabled() {
|
||||
let (window_x, window_y, window_width, window_height) =
|
||||
crate::wayland::window_geometry(xid)
|
||||
.ok_or_else(|| anyhow::anyhow!("No Wayland geometry for window {xid}"))?;
|
||||
if window_width == 0 || window_height == 0 {
|
||||
anyhow::bail!("Wayland window {xid} has no usable geometry");
|
||||
}
|
||||
let (screen_cx, screen_cy) =
|
||||
visible_element_point(bounds, (window_x, window_y, window_width, window_height))?;
|
||||
return Ok((
|
||||
xid,
|
||||
screen_cx - window_x as f64,
|
||||
@@ -1727,24 +1718,80 @@ fn screen_point_to_window_local(
|
||||
use x11rb::rust_connection::RustConnection;
|
||||
let (conn, screen_num) = RustConnection::connect(None)?;
|
||||
let root = conn.setup().roots[screen_num].root;
|
||||
if !crate::x11::window_belongs_to_pid(xid, pid) {
|
||||
anyhow::bail!("window {xid} no longer belongs to pid {pid}");
|
||||
}
|
||||
let reply = conn
|
||||
.translate_coordinates(xid as u32, root, 0, 0)?
|
||||
.reply()?;
|
||||
let geometry = conn.get_geometry(xid as u32)?.reply()?;
|
||||
let (screen_cx, screen_cy) = visible_element_point(
|
||||
bounds,
|
||||
(
|
||||
reply.dst_x.into(),
|
||||
reply.dst_y.into(),
|
||||
geometry.width.into(),
|
||||
geometry.height.into(),
|
||||
),
|
||||
)?;
|
||||
let local_x = screen_cx - reply.dst_x as f64;
|
||||
let local_y = screen_cy - reply.dst_y as f64;
|
||||
Ok((xid, local_x, local_y))
|
||||
}
|
||||
|
||||
fn visible_element_point(
|
||||
element: (i32, i32, u32, u32),
|
||||
window: (i32, i32, u32, u32),
|
||||
) -> anyhow::Result<(f64, f64)> {
|
||||
let left = i64::from(element.0).max(i64::from(window.0));
|
||||
let top = i64::from(element.1).max(i64::from(window.1));
|
||||
let right = (i64::from(element.0) + i64::from(element.2))
|
||||
.min(i64::from(window.0) + i64::from(window.2));
|
||||
let bottom = (i64::from(element.1) + i64::from(element.3))
|
||||
.min(i64::from(window.1) + i64::from(window.3));
|
||||
if right <= left || bottom <= top {
|
||||
anyhow::bail!("element has no visible area inside the target window");
|
||||
}
|
||||
Ok((
|
||||
(left + (right - left) / 2) as f64,
|
||||
(top + (bottom - top) / 2) as f64,
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[test]
|
||||
fn element_pointer_stays_inside_the_requested_window() {
|
||||
let window = (70, 102, 440, 300);
|
||||
assert_eq!(
|
||||
visible_element_point((90, 160, 700, 700), window).unwrap(),
|
||||
(300.0, 281.0)
|
||||
);
|
||||
assert!(visible_element_point((520, 410, 50, 50), window).is_err());
|
||||
assert!(visible_element_point((90, 160, 0, 40), window).is_err());
|
||||
assert_eq!(
|
||||
visible_element_point((509, 401, 100, 100), window).unwrap(),
|
||||
(509.0, 401.0)
|
||||
);
|
||||
assert_eq!(
|
||||
visible_element_point((-200, -200, 300, 300), (-100, -100, 50, 50)).unwrap(),
|
||||
(-75.0, -75.0)
|
||||
);
|
||||
}
|
||||
|
||||
fn element_screen_center_exact(
|
||||
pid: u32,
|
||||
idx: usize,
|
||||
xid_hint: Option<u64>,
|
||||
identity: Option<crate::atspi::AtspiIdentity>,
|
||||
) -> anyhow::Result<(f64, f64)> {
|
||||
let (bx, by, bw, bh) = match identity {
|
||||
Some(identity) => crate::atspi::get_element_bounds_exact(pid, idx, identity)?,
|
||||
None => crate::atspi::get_element_bounds(pid, idx)?,
|
||||
};
|
||||
Ok((bx as f64 + bw as f64 / 2.0, by as f64 + bh as f64 / 2.0))
|
||||
let (xid, x, y) = resolve_element_local_coords_exact(pid, idx, xid_hint, identity)?;
|
||||
if crate::wayland::wayland_input_enabled() {
|
||||
let (ox, oy, _, _) = crate::wayland::window_geometry(xid)
|
||||
.ok_or_else(|| anyhow::anyhow!("No Wayland geometry for window {xid}"))?;
|
||||
Ok((x + f64::from(ox), y + f64::from(oy)))
|
||||
} else {
|
||||
window_local_to_screen(xid, x, y)
|
||||
}
|
||||
}
|
||||
|
||||
fn cached_atspi_identity(
|
||||
@@ -1846,27 +1893,15 @@ fn non_ax_escalation() -> Value {
|
||||
/// `effect` tri-state. Linux's AT-SPI `insertText` return value acknowledges
|
||||
/// the method call but does not read the widget value back, so it and every
|
||||
/// keystroke / XSendEvent / XTest / Wayland rung are `"unverifiable"` (the
|
||||
/// caller confirms through a separate observation) and
|
||||
/// carries a `foreground` escalation, because the field IS in the AT-SPI tree —
|
||||
/// it's a delivery/focus problem, not a missing element. The foreground rung
|
||||
/// itself (`key_events_fg`) is already the last resort, so it emits no
|
||||
/// escalation. Mirrors the macOS `type_text` contract.
|
||||
/// caller confirms through a separate observation). An acknowledgement without
|
||||
/// readback does not establish failed delivery or require another input.
|
||||
fn type_text_structured(path: &str, characters: usize, verified: bool) -> Value {
|
||||
let mut s = json!({
|
||||
json!({
|
||||
"path": path,
|
||||
"characters": characters,
|
||||
"verified": verified,
|
||||
"effect": if verified { "confirmed" } else { "unverifiable" },
|
||||
});
|
||||
if !verified && path != "key_events_fg" {
|
||||
s["escalation"] = json!({
|
||||
"recommended": "foreground",
|
||||
"reason": "background insert could not be confirmed — re-call with \
|
||||
delivery_mode:\"foreground\" if a screenshot shows the text \
|
||||
didn't land."
|
||||
});
|
||||
}
|
||||
s
|
||||
})
|
||||
}
|
||||
|
||||
/// Structured payload for the Electron/Chromium AX-echo case: the AT-SPI
|
||||
@@ -2194,7 +2229,13 @@ fn x11_pixel_click_no_focus_steal(
|
||||
}
|
||||
|
||||
fn linux_input_error(error: anyhow::Error) -> ToolResult {
|
||||
if crate::input::is_uinput_unavailable(&error) {
|
||||
if crate::atspi::native::input_was_dispatched(&error) {
|
||||
ToolResult::error(format!("{error:#}")).with_structured(json!({
|
||||
"code": "atspi_input_unconfirmed",
|
||||
"effect": "unverifiable",
|
||||
"input_dispatched": true,
|
||||
}))
|
||||
} else if crate::input::is_uinput_unavailable(&error) {
|
||||
ToolResult::error(error.to_string()).with_structured(json!({
|
||||
"code": crate::input::UINPUT_UNAVAILABLE_CODE,
|
||||
}))
|
||||
@@ -2746,7 +2787,7 @@ impl Tool for ClickTool {
|
||||
// Previously perform_action ran inside this spawn_blocking, so the
|
||||
// app updated before the cursor visibly arrived.
|
||||
let (xid, sx, sy) = tokio::task::spawn_blocking(move || -> (u64, f64, f64) {
|
||||
let (cx, cy) = element_screen_center_exact(pid, idx, identity_for_center)
|
||||
let (cx, cy) = element_screen_center_exact(pid, idx, xid_hint, identity_for_center)
|
||||
.unwrap_or((0.0, 0.0));
|
||||
let xid = xid_hint
|
||||
.or_else(|| {
|
||||
@@ -2774,12 +2815,22 @@ impl Tool for ClickTool {
|
||||
|
||||
// Chromium can execute a genuine AT-SPI action without focus. Try
|
||||
// that route before applying its background synthetic-input gate.
|
||||
if modifiers.is_empty() {
|
||||
let ax_result = tokio::task::spawn_blocking(move || match exact_identity {
|
||||
if button == 1 && count == 1 && modifiers.is_empty() {
|
||||
let identity_for_action = exact_identity.clone();
|
||||
let ax_result = tokio::task::spawn_blocking(move || match identity_for_action {
|
||||
Some(identity) => crate::atspi::perform_action_exact(pid, idx, identity),
|
||||
None => crate::atspi::perform_action(pid, idx),
|
||||
})
|
||||
.await;
|
||||
let ax_result = match ax_result {
|
||||
Ok(Err(error)) if crate::atspi::native::input_was_dispatched(&error) => {
|
||||
return linux_input_error(error);
|
||||
}
|
||||
Err(error) => {
|
||||
return ToolResult::error(format!("AT-SPI action task failed: {error}"))
|
||||
}
|
||||
result => result,
|
||||
};
|
||||
if let Ok(Ok((_action, suspected_noop))) = ax_result {
|
||||
let mut structured = json!({
|
||||
"path": "ax",
|
||||
@@ -2800,7 +2851,8 @@ impl Tool for ClickTool {
|
||||
// The AX route was unavailable. Fall back to a target-addressed
|
||||
// X11 event for toolkits that accept it.
|
||||
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {
|
||||
let (xid2, lx, ly) = resolve_element_local_coords(pid, idx, xid_hint)?;
|
||||
let (xid2, lx, ly) =
|
||||
resolve_element_local_coords_exact(pid, idx, xid_hint, exact_identity.clone())?;
|
||||
let modifier_refs: Vec<&str> = modifiers.iter().map(String::as_str).collect();
|
||||
if crate::wayland::wayland_input_enabled() && !modifier_refs.is_empty() {
|
||||
anyhow::bail!(
|
||||
@@ -2815,6 +2867,8 @@ impl Tool for ClickTool {
|
||||
// clicks made those rows addressable but not selectable.
|
||||
if delivery.is_foreground() && !crate::wayland::wayland_input_enabled() {
|
||||
crate::input::with_x11_foreground(xid2, 80, || {
|
||||
let (sx, sy) =
|
||||
element_screen_center_exact(pid, idx, Some(xid2), exact_identity)?;
|
||||
crate::input::send_click_xtest_desktop_with_modifiers(
|
||||
sx.round() as i32,
|
||||
sy.round() as i32,
|
||||
@@ -2940,10 +2994,13 @@ impl Tool for ClickTool {
|
||||
// working. (x,y) are screen coords here, matching the frames in
|
||||
// `get_window_state`. Miss → fall through to the injection paths.
|
||||
if !delivery.is_foreground() && button == 1 && count == 1 {
|
||||
if let Ok(Some(_)) =
|
||||
crate::atspi::perform_action_at_screen_point(pid, xid, output_x, output_y)
|
||||
match crate::atspi::perform_action_at_screen_point(pid, xid, output_x, output_y)
|
||||
{
|
||||
return Ok("wayland_atspi");
|
||||
Ok(Some(_)) => return Ok("wayland_atspi"),
|
||||
Err(error) if crate::atspi::native::input_was_dispatched(&error) => {
|
||||
return Err(error)
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
if crate::wayland::is_inject_mode() {
|
||||
@@ -2968,8 +3025,12 @@ impl Tool for ClickTool {
|
||||
// click (the agent's escalation when background didn't land).
|
||||
let inject = |fg: bool| -> anyhow::Result<&'static str> {
|
||||
if !fg && button == 1 && count == 1 && modifiers_for_task.is_empty() {
|
||||
if let Ok(Some(_)) = crate::atspi::perform_action_at_point(pid, xi, yi) {
|
||||
return Ok("x11_atspi");
|
||||
match crate::atspi::perform_action_at_point(pid, xi, yi) {
|
||||
Ok(Some(_)) => return Ok("x11_atspi"),
|
||||
Err(error) if crate::atspi::native::input_was_dispatched(&error) => {
|
||||
return Err(error)
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
if fg {
|
||||
@@ -3291,7 +3352,7 @@ impl Tool for TypeTextTool {
|
||||
);
|
||||
let identity_for_center = exact_identity.clone();
|
||||
if let Ok(Ok((screen_x, screen_y))) = tokio::task::spawn_blocking(move || {
|
||||
element_screen_center_exact(pid, idx, identity_for_center)
|
||||
element_screen_center_exact(pid, idx, Some(xid), identity_for_center)
|
||||
})
|
||||
.await
|
||||
{
|
||||
@@ -3319,8 +3380,17 @@ impl Tool for TypeTextTool {
|
||||
type_into_atspi_target(pid, idx, identity_for_type, &text_at)
|
||||
})
|
||||
.await;
|
||||
if let Ok(Ok(())) = targeted {
|
||||
return type_text_ax_result(pid, text_len, "via targeted AT-SPI");
|
||||
match targeted {
|
||||
Ok(Ok(())) => {
|
||||
return type_text_ax_result(pid, text_len, "via targeted AT-SPI");
|
||||
}
|
||||
Ok(Err(error)) if crate::atspi::native::input_was_dispatched(&error) => {
|
||||
return linux_input_error(error);
|
||||
}
|
||||
Err(error) => {
|
||||
return ToolResult::error(format!("AT-SPI typing task failed: {error}"))
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
// The private nested compositor can target the owning Wayland client
|
||||
@@ -3443,13 +3513,20 @@ impl Tool for TypeTextTool {
|
||||
Ok(Ok(())) => {
|
||||
return type_text_ax_result(pid, text_len, "via targeted AT-SPI");
|
||||
}
|
||||
Ok(Err(_)) | Err(_)
|
||||
Ok(Err(error)) if crate::atspi::native::input_was_dispatched(&error) => {
|
||||
return linux_input_error(error);
|
||||
}
|
||||
Err(error) => {
|
||||
return ToolResult::error(format!("AT-SPI typing task failed: {error}"))
|
||||
}
|
||||
Ok(Err(_))
|
||||
if !delivery.is_foreground() && crate::wayland::wayland_input_enabled() =>
|
||||
{
|
||||
return crate::input::delivery::background_unavailable_error(
|
||||
crate::input::delivery::BackgroundUnavailable::FocusedInputOnly,
|
||||
);
|
||||
}
|
||||
Ok(Err(error)) if !delivery.is_foreground() => return linux_input_error(error),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
@@ -3545,7 +3622,11 @@ impl Tool for TypeTextTool {
|
||||
// and WebKitGTK can acknowledge an accessibility write without
|
||||
// producing the renderer input event, so web embedders use real XTest
|
||||
// key events. Native toolkits keep their verifiable AT-SPI path below.
|
||||
if delivery.is_foreground() && (is_chromium_embedder(pid) || is_webkitgtk_embedder(pid)) {
|
||||
if delivery.is_foreground()
|
||||
&& (resolved_elem_idx.is_some()
|
||||
|| is_chromium_embedder(pid)
|
||||
|| is_webkitgtk_embedder(pid))
|
||||
{
|
||||
let text_f = text.clone();
|
||||
let idx = resolved_elem_idx;
|
||||
let identity_for_focus = exact_identity.clone();
|
||||
@@ -3637,6 +3718,10 @@ impl Tool for TypeTextTool {
|
||||
// observing it, so the confirm is suppressed there (mirrors macOS).
|
||||
return type_text_ax_result(pid, text_len, "via AT-SPI");
|
||||
}
|
||||
Ok(Err(error)) if crate::atspi::native::input_was_dispatched(&error) => {
|
||||
return linux_input_error(error);
|
||||
}
|
||||
Err(error) => return ToolResult::error(format!("AT-SPI typing task failed: {error}")),
|
||||
_ => {
|
||||
// AT-SPI failed (no editable exposed). Qt5 doesn't expose widgets
|
||||
// when unfocused, so try the synthetic-focus workaround.
|
||||
@@ -3656,16 +3741,25 @@ impl Tool for TypeTextTool {
|
||||
let result = crate::atspi::type_into_editable(pid, &text_clone2);
|
||||
|
||||
// Restore state with FocusOut
|
||||
crate::input::send_focus_out(xid)?;
|
||||
let restore = crate::input::send_focus_out(xid);
|
||||
|
||||
result
|
||||
Ok::<_, anyhow::Error>((result, restore))
|
||||
})
|
||||
.await;
|
||||
|
||||
match qt5_result {
|
||||
Ok(Ok(())) => {
|
||||
Ok(Ok((Ok(()), Ok(())))) => {
|
||||
return type_text_ax_result(pid, text_len, "via AT-SPI with focus workaround");
|
||||
}
|
||||
Ok(Ok((Err(error), _))) if crate::atspi::native::input_was_dispatched(&error) => {
|
||||
return linux_input_error(error);
|
||||
}
|
||||
Ok(Ok((_, Err(error)))) => {
|
||||
return ToolResult::error(format!(
|
||||
"AT-SPI typing focus restoration failed: {error}; observe before retrying"
|
||||
));
|
||||
}
|
||||
Err(error) => return ToolResult::error(format!("AT-SPI typing task failed: {error}")),
|
||||
_ => {
|
||||
// AT-SPI still didn't work. Fall back to X11 XSendEvent.
|
||||
}
|
||||
@@ -3688,8 +3782,12 @@ impl Tool for TypeTextTool {
|
||||
// focused widget, so background XSendEvent typing doesn't land. Fill
|
||||
// the editable field via AT-SPI instead — focus-free and toolkit-
|
||||
// agnostic. Fall back to Tk send or XSendEvent when no a11y field is exposed.
|
||||
if crate::atspi::insert_text(pid, &text).unwrap_or(false) {
|
||||
return Ok("ax");
|
||||
match crate::atspi::insert_text(pid, &text) {
|
||||
Ok(true) => return Ok("ax"),
|
||||
Err(error) if crate::atspi::native::input_was_dispatched(&error) => {
|
||||
return Err(error)
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
// Tk apps: use Tk's `send` command (no AT-SPI bridge, so AT-SPI above
|
||||
// returned false). This is the Tk-specific override, like CDP for Chromium.
|
||||
@@ -3729,7 +3827,7 @@ impl Tool for TypeTextTool {
|
||||
"Typed {text_len} character(s) (via X11, delivery_mode={mode_label})."
|
||||
))
|
||||
.with_structured(type_text_structured(path, text_len, false)),
|
||||
Ok(Err(e)) => ToolResult::error(e.to_string()),
|
||||
Ok(Err(e)) => linux_input_error(e),
|
||||
Err(e) => ToolResult::error(format!("Task error: {e}")),
|
||||
}
|
||||
}
|
||||
@@ -4537,7 +4635,7 @@ impl Tool for SetValueTool {
|
||||
// be resolved or the overlay is disabled.
|
||||
let identity_for_center = exact_identity.clone();
|
||||
if let Ok(Ok((sx, sy))) = tokio::task::spawn_blocking(move || {
|
||||
element_screen_center_exact(pid, idx, identity_for_center)
|
||||
element_screen_center_exact(pid, idx, Some(xid), identity_for_center)
|
||||
})
|
||||
.await
|
||||
{
|
||||
@@ -4557,7 +4655,7 @@ impl Tool for SetValueTool {
|
||||
.await;
|
||||
match result {
|
||||
Ok(Ok(())) => ToolResult::text(format!("Set value of element [{idx}] to '{value}'.")),
|
||||
Ok(Err(e)) => ToolResult::error(e.to_string()),
|
||||
Ok(Err(e)) => linux_input_error(e),
|
||||
Err(e) => ToolResult::error(format!("Task error: {e}")),
|
||||
}
|
||||
}
|
||||
@@ -4857,6 +4955,15 @@ impl Tool for ScrollTool {
|
||||
None => crate::atspi::scroll_element(pid, idx, &direction_for_ax, amount),
|
||||
})
|
||||
.await;
|
||||
let ax_result = match ax_result {
|
||||
Ok(Err(error)) if crate::atspi::native::input_was_dispatched(&error) => {
|
||||
return linux_input_error(error);
|
||||
}
|
||||
Err(error) => {
|
||||
return ToolResult::error(format!("AT-SPI scroll task failed: {error}"))
|
||||
}
|
||||
result => result,
|
||||
};
|
||||
if matches!(ax_result, Ok(Ok(()))) {
|
||||
let mode = if delivery.is_foreground() {
|
||||
"foreground"
|
||||
@@ -5024,6 +5131,12 @@ impl Tool for ScrollTool {
|
||||
let cursor_id_for_task = cursor_id.clone();
|
||||
let direction_for_wayland = direction.clone();
|
||||
let amount_u32 = amount as u32;
|
||||
let element_index = match &resolved {
|
||||
cua_driver_core::element_token::ResolvedElement::Element { element_index, .. } => {
|
||||
Some(*element_index)
|
||||
}
|
||||
_ => None,
|
||||
};
|
||||
let result = tokio::task::spawn_blocking(move || -> anyhow::Result<()> {
|
||||
if crate::wayland::wayland_input_enabled() {
|
||||
return crate::wayland::scroll(xid, &direction_for_wayland, amount_u32);
|
||||
@@ -5072,15 +5185,19 @@ impl Tool for ScrollTool {
|
||||
crate::input::send_click(xid, local_x as i32, local_y as i32, amount, button)
|
||||
};
|
||||
if delivery.is_foreground() {
|
||||
let point = element_point
|
||||
.and_then(|(_, screen)| screen)
|
||||
.or_else(|| {
|
||||
window_screen_center(xid)
|
||||
.ok()
|
||||
.map(|(x, y)| (x as f64, y as f64))
|
||||
})
|
||||
.ok_or_else(|| anyhow::anyhow!("could not resolve foreground scroll point"))?;
|
||||
crate::input::with_x11_foreground(xid, 80, || {
|
||||
let point = match element_index {
|
||||
Some(idx) => {
|
||||
element_screen_center_exact(pid, idx, Some(xid), exact_identity)?
|
||||
}
|
||||
None => match element_point {
|
||||
Some(((x, y), _)) => window_local_to_screen(xid, x, y)?,
|
||||
None => {
|
||||
let (x, y) = window_screen_center(xid)?;
|
||||
(f64::from(x), f64::from(y))
|
||||
}
|
||||
},
|
||||
};
|
||||
crate::input::send_click_xtest_desktop(
|
||||
point.0 as i32,
|
||||
point.1 as i32,
|
||||
@@ -5215,8 +5332,9 @@ impl Tool for DoubleClickTool {
|
||||
.await;
|
||||
return match result {
|
||||
Ok(Ok((xid, lx, ly))) => {
|
||||
let identity_for_overlay = exact_identity.clone();
|
||||
if let Ok(Ok((sx, sy))) = tokio::task::spawn_blocking(move || {
|
||||
element_screen_center_exact(pid, idx, exact_identity)
|
||||
element_screen_center_exact(pid, idx, Some(xid), identity_for_overlay)
|
||||
})
|
||||
.await
|
||||
{
|
||||
@@ -5245,7 +5363,12 @@ impl Tool for DoubleClickTool {
|
||||
}
|
||||
if delivery.is_foreground() {
|
||||
return crate::input::with_x11_foreground(xid, 80, || {
|
||||
let (sx, sy) = window_local_to_screen(xid, lxi as f64, lyi as f64)?;
|
||||
let (sx, sy) = element_screen_center_exact(
|
||||
pid,
|
||||
idx,
|
||||
Some(xid),
|
||||
exact_identity,
|
||||
)?;
|
||||
crate::input::send_click_xtest_desktop(
|
||||
sx.round() as i32,
|
||||
sy.round() as i32,
|
||||
@@ -5469,8 +5592,9 @@ impl Tool for RightClickTool {
|
||||
.await;
|
||||
return match result {
|
||||
Ok(Ok((xid, lx, ly))) => {
|
||||
let identity_for_overlay = exact_identity.clone();
|
||||
if let Ok(Ok((sx, sy))) = tokio::task::spawn_blocking(move || {
|
||||
element_screen_center_exact(pid, idx, exact_identity)
|
||||
element_screen_center_exact(pid, idx, Some(xid), identity_for_overlay)
|
||||
})
|
||||
.await
|
||||
{
|
||||
@@ -5499,7 +5623,12 @@ impl Tool for RightClickTool {
|
||||
}
|
||||
if delivery.is_foreground() {
|
||||
return crate::input::with_x11_foreground(xid, 80, || {
|
||||
let (sx, sy) = window_local_to_screen(xid, lxi as f64, lyi as f64)?;
|
||||
let (sx, sy) = element_screen_center_exact(
|
||||
pid,
|
||||
idx,
|
||||
Some(xid),
|
||||
exact_identity,
|
||||
)?;
|
||||
crate::input::send_click_xtest_desktop(
|
||||
sx.round() as i32,
|
||||
sy.round() as i32,
|
||||
|
||||
@@ -523,7 +523,7 @@ done
|
||||
# asset — see the recovery at the download step below.
|
||||
#
|
||||
# ~~~ BAKED_VERSION: must match the release version in this change ~~~
|
||||
CUA_DRIVER_RS_BAKED_VERSION="0.20.8"
|
||||
CUA_DRIVER_RS_BAKED_VERSION="0.20.9"
|
||||
# ~~~ END_BAKED_VERSION ~~~
|
||||
|
||||
# Run API requests with an optional token. Keep the header construction here
|
||||
|
||||
@@ -118,7 +118,7 @@ $ThemeBinaryName = "cua-cursor-theme.exe"
|
||||
# where the baked line hasn't been updated yet.
|
||||
#
|
||||
# ~~~ BAKED_VERSION: must match the release version in this change ~~~
|
||||
$Script:CuaDriverRsBakedVersion = "0.20.8"
|
||||
$Script:CuaDriverRsBakedVersion = "0.20.9"
|
||||
# ~~~ END_BAKED_VERSION ~~~
|
||||
$CursorThemeRequiredFrom = [version]"0.12.7"
|
||||
$LegacyUiaRequiredFrom = [version]"0.2.8"
|
||||
|
||||
@@ -16,8 +16,8 @@
|
||||
# --backend=swift retired Swift backend (no-op; accepted for compat)
|
||||
#
|
||||
# Env overrides:
|
||||
# CUA_DRIVER_RS_VERSION=0.20.8 pin a specific Rust release tag
|
||||
# CUA_DRIVER_VERSION=0.20.8 legacy alias for CUA_DRIVER_RS_VERSION
|
||||
# CUA_DRIVER_RS_VERSION=0.20.9 pin a specific Rust release tag
|
||||
# CUA_DRIVER_VERSION=0.20.9 legacy alias for CUA_DRIVER_RS_VERSION
|
||||
# CUA_DRIVER_RS_INSTALL_DIR=PATH same as --bin-dir
|
||||
# CUA_DRIVER_BIN_DIR=PATH legacy alias for --bin-dir
|
||||
# CUA_DRIVER_NO_MODIFY_PATH=1 same as --no-modify-path
|
||||
|
||||
@@ -1031,6 +1031,8 @@ export class ComputerUse {
|
||||
}
|
||||
if (
|
||||
!retriedIncompleteCapture &&
|
||||
!capture.incompleteDetails.some((detail) =>
|
||||
detail === "walk_deadline_reached" || detail === "element_bounds_timeout") &&
|
||||
capture.complete === false &&
|
||||
(!capture.truncated || capture.readComplete === false) &&
|
||||
envelope?.stable_element_ids !== true &&
|
||||
|
||||
@@ -450,7 +450,6 @@ for (const [readComplete, details] of [
|
||||
[undefined, ["AXTitle: ax_error -25204", "walk: max_elements truncated"]],
|
||||
[false, ["walk: max_elements truncated"]],
|
||||
[undefined, ["provider_unresponsive", "max_elements_reached"]],
|
||||
[undefined, ["walk_deadline_reached", "max_depth_reached"]],
|
||||
]) {
|
||||
test(`mixed read failure and truncation retries once (read flag ${readComplete})`, async () => {
|
||||
let reads = 0;
|
||||
@@ -487,6 +486,49 @@ for (const [readComplete, details] of [
|
||||
});
|
||||
}
|
||||
|
||||
for (const reason of ["walk_deadline_reached", "element_bounds_timeout"]) {
|
||||
for (const nested of [false, true]) {
|
||||
test(`deadline capture returns its prefix without another walk (${reason}, nested ${nested})`, async () => {
|
||||
const complete = (id) => toolResult({ structured: {
|
||||
tree_markdown: "Ready",
|
||||
elements: [{ element_token: `rv1:${id}:0` }],
|
||||
capture_complete: true,
|
||||
observation_revision: {
|
||||
mode: "full", lineage_id: id, revision_id: `${id}:r1`, stable_element_ids: true,
|
||||
},
|
||||
} });
|
||||
const capture = {
|
||||
capture_complete: false,
|
||||
capture_truncated: true,
|
||||
capture_incomplete_details: [reason, "max_depth_reached"],
|
||||
};
|
||||
const results = [complete("old"), toolResult({ structured: {
|
||||
...(nested ? {} : capture),
|
||||
tree_markdown: "Completed prefix",
|
||||
elements: [{ element_index: 0, element_token: "s00000001:0" }],
|
||||
observation_revision: {
|
||||
...(nested ? capture : {}),
|
||||
mode: "full", revision_id: "transient:r0", stable_element_ids: false,
|
||||
resync_reason: "capture_incomplete",
|
||||
},
|
||||
} }), complete("recovered")];
|
||||
const driver = fakeDriver({ results: { getWindowState: () => results.shift() } });
|
||||
const computer = new ComputerUse(driver, { sdk: fakeSdk });
|
||||
await computer.observeWindow({ pid: 42, windowId: 7 });
|
||||
const partial = await computer.observeWindow({ pid: 42, windowId: 7 });
|
||||
assert.equal(driver.calls.length, 2);
|
||||
assert.match(partial.text, /Completed prefix/);
|
||||
assert.equal(partial.elements[0].element_token, "s00000001:0");
|
||||
assert.equal(partial.diagnostics.captureReadComplete, false);
|
||||
assert.equal(partial.diagnostics.stableElementIds, false);
|
||||
const recovered = await computer.observeWindow({ pid: 42, windowId: 7 });
|
||||
assert.equal(driver.calls.length, 3);
|
||||
assert.equal(driver.calls[2].input.observationRevision.baseRevisionId, undefined);
|
||||
assert.equal(recovered.diagnostics.stableElementIds, true);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
for (const detail of ["max_elements_reached", "max_depth_reached"]) {
|
||||
test(`legacy Linux budget capture is not retried: ${detail}`, async () => {
|
||||
const driver = fakeDriver({ results: {
|
||||
|
||||
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@qwen-code/cua-sdk",
|
||||
"version": "0.20.8",
|
||||
"version": "0.20.9",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@qwen-code/cua-sdk",
|
||||
"version": "0.20.8",
|
||||
"version": "0.20.9",
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@qwen-code/cua-sdk",
|
||||
"version": "0.20.8",
|
||||
"version": "0.20.9",
|
||||
"description": "Typed CUA Driver SDK and Computer Use API for Node.js",
|
||||
"type": "module",
|
||||
"license": "MIT",
|
||||
|
||||
Reference in new issue
Block a user