Compare commits

...
Author SHA1 Message Date
Brent Vatne fecf040ce5 Release 0.20.3 (#53) 2026-10-20 12:22:14 -07:00
devin-ai-integration[bot]andbrent e47399440f Store opened iPhone Duo screenshots upright in explore:binary/session (#51)
Co-authored-by: brent <[email protected]>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 12:18:17 -07:00
devin-ai-integration[bot]andbrent 46b46bb184 Add --replay to explore:binary and explore:session (#50)
Co-authored-by: brent <[email protected]>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 12:11:32 -07:00
Brent Vatne 942bdaad40 Capture waits for a focused field's keyboard, and the comment names off-screen changes (#52) 2026-10-20 12:09:40 -07:00
Brent Vatne ed31b0c767 Release 0.20.2 (#49) 2026-10-19 19:50:28 -07:00
38 changed files with 1756 additions and 101 deletions

No files matched your search

+35
View File
@@ -198,6 +198,41 @@ npx @expo/explore explore:session <session-id> --env-file path/.env.eas-simulato
</details>
<details>
<summary><strong>Take the same steps on every run and device</strong></summary>
A model picks its own path, so two runs (or an iPhone and an Android phone) can reach different screens. To get the same screens every time, write the steps down once, for example after a model run showed you the app, and pass them with `--replay`. No model login or API key is needed.
```sh
npx @expo/explore explore:binary ./my-app.app --replay steps.json
npx @expo/explore explore:session --replay steps.json # an iPhone Duo pose, or any session you set up by hand
```
```json
{
"steps": [
{ "capture": "Home" },
{ "action": "tap", "label": "Settings" },
{ "capture": "Settings" },
{ "action": "tap", "label": "Dark" },
{ "capture": "Settings", "state": "dark" },
{ "action": "tap", "label": "Learn", "platforms": ["ios"] },
{ "capture": "Learn", "platforms": ["ios"] },
{ "action": "tap", "label": "Got it", "optional": true }
]
}
```
- An action is one of `tap` or `fill` (with `label` or `identifier`), `type`, `scroll` (`direction`), `wait` (`milliseconds`), `open` (`url`), `back`, and `dismiss-keyboard`, as in the model's own tools. The same safety rules apply: destructive or app-leaving taps are refused. A step takes only the fields listed here; any other field (a typo such as `platform`) is an error.
- `{ "capture": "Title" }` photographs the current screen under that title; add `"state"` for another state of a screen already captured. A capture of a screen the plan already captured under another title is reported as looking the same, and the screen keeps its first title.
- `"platforms": ["ios"]` limits a step to those platforms, for screens one platform does not have. `"optional": true` skips an action that fails (a one-time dialog) instead of stopping; the summary lists the steps it skipped.
- A `fill` or `type` text can be a `{{secret:NAME}}`, as in `--hints`: Explore types the value of the environment variable `NAME`, and a variable that is not set fails before a device starts.
- If a required action fails, the run stops there and the report still shows what was captured, with a gap naming the step and the screens it did not reach.
- The screen and action budgets still apply. A plan with more captures or actions than they allow is refused before a device starts, as is one with no steps for the platform; raise `--max-screens` and `--max-actions` for a long plan. If a budget still runs out (each screen the plan reaches also counts), the gap names the planned screens not reached.
- `--provider`, `--model` and `--hints` only steer a model, so they cannot be combined with `--replay`.
</details>
<details>
<summary><strong>What a run without source shows</strong></summary>
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@expo/explore",
"version": "0.20.1",
"version": "0.20.3",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@expo/explore",
"version": "0.20.1",
"version": "0.20.3",
"license": "MIT",
"dependencies": {
"@expo/repack-app": "0.10.1",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@expo/explore",
"version": "0.20.1",
"version": "0.20.3",
"description": "Source-driven mobile app maps, deterministic CI capture, and visual reviews",
"license": "MIT",
"type": "module",
+9
View File
@@ -202,6 +202,15 @@
"mimeType": { "type": "string", "minLength": 1 },
"width": { "type": "integer", "minimum": 0 },
"height": { "type": "integer", "minimum": 0 },
"turned": {
"type": "object",
"additionalProperties": false,
"required": ["width", "height"],
"properties": {
"width": { "type": "number", "exclusiveMinimum": 0 },
"height": { "type": "number", "exclusiveMinimum": 0 }
}
},
"createdAt": { "$ref": "#/$defs/timestamp" }
}
},
+4
View File
@@ -33,6 +33,10 @@ Platform, bundle id, app name and version are read from the archive; do not pass
Budgets default to 15 minutes, 40 screens, 150 actions (`--max-duration-minutes`, `--max-screens`, `--max-actions`). Steer the model with `--hints "focus on the onboarding and settings"`, skip onboarding with `--launch-url myapp://home`, and allow a specific risky control with `--allow-destructive "Clear cache"`.
## The same steps on every run: `--replay`
To compare runs, or an iPhone with an Android phone, screen for screen, pass `--replay steps.json` to `explore:binary` or `explore:session`: Explore takes the steps in the file instead of a model's, and no model access is needed. The format is in the README ("Take the same steps on every run and device"): `{ "steps": [{ "capture": "Home" }, { "action": "tap", "label": "Settings" }, { "capture": "Settings" }] }`, with `platforms` to limit a step and `optional` for an action that may fail. The plan is checked before a device starts: unknown fields, a plan with no steps for the platform, more captures or actions than `--max-screens` / `--max-actions`, and a `{{secret:NAME}}` in a `fill` or `type` whose variable is not set are all refused (`REPLAY_PLAN_INVALID`, `REPLAY_PLAN_EMPTY`, `REPLAY_PLAN_OVER_BUDGET`, `EXPLORATION_SECRET_MISSING`). `--provider`, `--model` and `--hints` cannot be combined with it (`REPLAY_OPTIONS_IGNORED`).
## Sign-in and other app state
Without help, the model treats a sign-in wall as the end of the road: it captures it and stops (`blocked`). Before running, read what you can about the app (its source if you have it, its first screens otherwise) and decide what it needs.
+59 -27
View File
@@ -7,6 +7,7 @@ import type { CaptureSetupAction } from "../config/explore-config.js";
import { createExplorationInputs, explorationSecretNames } from "../config/exploration-inputs.js";
import type { ExploreStore, PreparedArtifact } from "../map/store.js";
import { CiError } from "../shared/ci-errors.js";
import { turnedClockwise } from "../shared/png-evidence.js";
import type { ActionKind, ActionRisk, Artifact, Controller, ElementRole, ExploreMap, ObservedElement, Observation, Screen } from "../types.js";
/** One look at the device: accessibility always, a PNG when the host could take a safe one. */
@@ -21,13 +22,15 @@ export type BlackBoxFrame = {
imageUnavailable?: { code: string; message: string };
/** The foreground app differed from the explored one; the host already brought it back. */
leftApp?: boolean;
/** The image is on a portrait panel's axes under a landscape UI (an opened iPhone Duo), as the tree is; the report stores it turned upright. */
landscape?: boolean;
};
export type BlackBoxAction = CaptureSetupAction | { action: "back" } | { action: "dismiss-keyboard" } | { action: "type"; text: string }
| { action: "tap"; frameId: string; x: number; y: number };
export type BlackBoxExplorerOptions = {
session: CiAgentSession;
session: Pick<CiAgentSession, "run">;
store: ExploreStore;
map: ExploreMap;
app: { name: string; bundleId: string; platform: "ios" | "android"; version?: string };
@@ -49,6 +52,8 @@ export type BlackBoxExplorerOptions = {
appIcon?: Buffer;
/** App-test inputs the operator hints reference as {{secret:NAME}}: the model and the map only ever hold the reference. */
inputs?: ReturnType<typeof createExplorationInputs>;
/** A sentence a budget stop's gap adds to its own, read once the session has stopped: a replay names the planned screens it did not reach. */
budgetNote?: () => string | undefined;
};
export type BlackBoxExplorerResult = {
@@ -67,6 +72,12 @@ const EXTERNAL = /\b(open in|safari|chrome|browser|app store|play store|share|ma
const VOLATILE = /\d|[:%$€£]|\b(am|pm)\b/gi;
const INTERACTIVE = new Set(["button", "link", "cell", "switch", "tab", "menuitem", "textfield", "securetextfield", "searchfield", "textview", "edittext", "slider", "checkbox", "radiobutton", "segmentedcontrol", "toggle"]);
/** What the tools accept: their schemas, the checks on each call, and a replay plan checked before any device starts share these. */
export const TOOL_LIMITS = { selector: 512, text: 2000, url: 2000, waitMs: 15_000, title: 80, state: 80, description: 500, summary: 4000, reason: 1000 } as const;
/** A screen seen after an action but never named keeps this title until the run ends and names it after the path to it. */
export const isPlaceholderTitle = (title: string): boolean => /^Screen \d+$/.test(title);
const selectorProperties = {
label: { type: "string", description: "Exact visible label of the target node, copied from the latest observation." },
identifier: { type: "string", description: "Exact accessibility identifier of the target node, when it has one." },
@@ -81,22 +92,22 @@ const tools: CiAgentTool[] = [
inputSchema: { type: "object", properties: {
action: { type: "string", enum: ["tap", "fill", "type", "scroll", "wait", "back", "dismiss-keyboard", "open"] },
...selectorProperties,
text: { type: "string", maxLength: 2000 }, url: { type: "string", maxLength: 2000 },
text: { type: "string", maxLength: TOOL_LIMITS.text }, url: { type: "string", maxLength: TOOL_LIMITS.url },
direction: { type: "string", enum: ["up", "down", "left", "right", "top", "bottom"] },
milliseconds: { type: "integer", minimum: 1, maximum: 15_000 },
milliseconds: { type: "integer", minimum: 1, maximum: TOOL_LIMITS.waitMs },
frameId: { type: "string", description: "For a coordinate tap only: the latest observed frame." },
x: { type: "number", minimum: 0, exclusiveMaximum: 1 }, y: { type: "number", minimum: 0, exclusiveMaximum: 1 },
target: { type: "string", minLength: 1, maxLength: 80, description: "For a coordinate tap: what is being tapped, as a user would name it (\"search icon in the header\", \"first manga cover\")." },
}, required: ["action"], additionalProperties: false } },
{ name: "capture", description: "Name the latest observed frame (observe with includeImage) as a screen of the map, with a short human title. A screen is photographed the moment it first appears; capture gives it its title, and with state records another notable state of it.",
inputSchema: { type: "object", properties: {
frameId: { type: "string" }, title: { type: "string", minLength: 1, maxLength: 80 },
description: { type: "string", maxLength: 500 }, state: { type: "string", maxLength: 80, description: "A variant of an already captured screen, such as 'empty' or 'menu open'." },
frameId: { type: "string" }, title: { type: "string", minLength: 1, maxLength: TOOL_LIMITS.title },
description: { type: "string", maxLength: TOOL_LIMITS.description }, state: { type: "string", maxLength: TOOL_LIMITS.state, description: "A variant of an already captured screen, such as 'empty' or 'menu open'." },
}, required: ["frameId", "title"], additionalProperties: false } },
{ name: "finish", description: "End the exploration when every reachable screen has been captured or the budget is nearly used. Include a summary of what the app does and what was not reached.",
inputSchema: { type: "object", properties: { summary: { type: "string", minLength: 1, maxLength: 4000 } }, required: ["summary"], additionalProperties: false } },
inputSchema: { type: "object", properties: { summary: { type: "string", minLength: 1, maxLength: TOOL_LIMITS.summary } }, required: ["summary"], additionalProperties: false } },
{ name: "stop_blocked", description: "Stop because the app cannot be explored further (login wall without credentials, crash loop, permission prompt that cannot be dismissed). Keeps the evidence so far.",
inputSchema: { type: "object", properties: { reason: { type: "string", minLength: 1, maxLength: 1000 } }, required: ["reason"], additionalProperties: false } },
inputSchema: { type: "object", properties: { reason: { type: "string", minLength: 1, maxLength: TOOL_LIMITS.reason } }, required: ["reason"], additionalProperties: false } },
];
const INSTRUCTIONS = `You are exploring a prebuilt mobile app on a remote simulator through accessibility and screenshots. Nobody gave you its source. Your job is to produce a map of its screens: visit every reachable screen, capture each one once with a clear title, and try each distinct kind of control so transitions are recorded.
@@ -412,7 +423,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
}
/** Seen after an action but never named by the model (its screenshot was taken on arrival). */
const isPlaceholder = (screen: Screen): boolean => /^Screen \d+$/.test(screen.title);
const isPlaceholder = (screen: Screen): boolean => isPlaceholderTitle(screen.title);
/** Fold an uncaptured placeholder into the screen the model says it is. */
const mergeScreens = (from: Screen, into: Screen): void => {
for (const item of map.actionAttempts) {
@@ -437,8 +448,22 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
if (currentScreenId === from.id) currentScreenId = into.id;
};
const screenWithImage = (png: Buffer, except: Screen): Screen | undefined => {
const sha = createHash("sha256").update(png).digest("hex");
/**
* The bytes a frame's screenshot is stored as, and their digest: an opened Duo's frame is turned upright once, however
* often the frame is looked up or recorded. Keyed by the image, which every copy of the frame shares.
*/
const storedImages = new WeakMap<Buffer, { png: Buffer; sha256: string }>();
const storedImage = (image: Buffer, landscape: boolean): { png: Buffer; sha256: string } => {
let stored = storedImages.get(image);
if (!stored) {
const png = landscape ? turnedClockwise(image) : image;
stored = { png, sha256: createHash("sha256").update(png).digest("hex") };
storedImages.set(image, stored);
}
return stored;
};
const screenWithImage = (sha: string, except: Screen): Screen | undefined => {
const artifact = map.artifacts.find(item => item.kind === "screenshot" && item.sha256 === sha);
if (!artifact) return undefined;
const controls = structural(evidence.get(except.id)?.parts ?? new Set());
@@ -491,7 +516,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
screen = { id: `scr_${key.slice(0, 16)}`, title: title ?? systemDialogTitle(frame.nodes) ?? `Screen ${map.screens.length + 1}`, start: map.screens.length === 0, terminal: false,
depth, firstSeenAt: frame.capturedAt, lastSeenAt: frame.capturedAt, visits: 0, variantIds: [], observationIds: [], elementIds: [] };
map.screens.push(screen); screensByKey.set(key, screen);
} else if (title && /^Screen \d+$/.test(screen.title)) screen.title = title;
} else if (title && isPlaceholderTitle(screen.title)) screen.title = title;
screen.visits += 1; screen.lastSeenAt = frame.capturedAt;
const seen = evidence.get(screen.id) ?? { parts: new Set<string>(), loadingOnly: true };
for (const part of loose) seen.parts.add(part);
@@ -529,11 +554,13 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
let screenshotArtifactId: string | undefined;
if (withImage && frame.png) {
// Hashed here rather than trusting the daemon's digest: the report validates artifacts against their bytes.
const png = frame.png, sha = createHash("sha256").update(png).digest("hex");
const { png, sha256: sha } = storedImage(frame.png, frame.landscape === true);
const extent = frame.landscape ? frameExtent(frame.nodes) : undefined;
let artifact = map.artifacts.find(item => item.kind === "screenshot" && item.sha256 === sha);
if (!artifact) {
artifact = { id: `art_${randomUUID().slice(0, 12)}`, kind: "screenshot" as const, path: `artifacts/screenshots/${sha}.png`, sha256: sha,
bytes: png.length, mimeType: "image/png", ...(frame.width ? { width: frame.width } : {}), ...(frame.height ? { height: frame.height } : {}), createdAt: frame.capturedAt };
bytes: png.length, mimeType: "image/png", ...(frame.width && frame.height ? frame.landscape ? { width: frame.height, height: frame.width } : { width: frame.width, height: frame.height } : {}),
...(extent ? { turned: extent } : {}), createdAt: frame.capturedAt };
map.artifacts.push(artifact); prepared.push({ artifact, bytes: png, destination: join(store.root, artifact.path) });
}
screenshotArtifactId = artifact.id;
@@ -599,7 +626,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
// The same pixels as a screen already seen, with the same controls, is that screen: a tree read while Android was
// still exposing a text node (AudioScape's playlist title missing from one read of two identical frames) is not new.
if (created && latest.png) {
const twin = screenWithImage(latest.png, screen);
const twin = screenWithImage(storedImage(latest.png, latest.landscape === true).sha256, screen);
if (twin) { mergeScreens(screen, twin); screen = twin; created = false; }
}
// Every screen gets an observation the moment it is seen, so elements and action attempts always point at a
@@ -625,7 +652,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
const actionFor = (input: Record<string, unknown>): { action: BlackBoxAction; kind: ActionKind; label: string; risk: ActionRisk } => {
const action = String(input.action);
const selector = input.identifier !== undefined ? { identifier: text(input.identifier, "identifier", 512) } : input.label !== undefined ? { label: text(input.label, "label", 512) } : undefined;
const selector = input.identifier !== undefined ? { identifier: text(input.identifier, "identifier", TOOL_LIMITS.selector) } : input.label !== undefined ? { label: text(input.label, "label", TOOL_LIMITS.selector) } : undefined;
const label = selector ? ("identifier" in selector ? selector.identifier : selector.label) : action;
const risk = selector ? riskOf(label, allow) : "safe";
switch (action) {
@@ -637,13 +664,13 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
}
if (!selector) throw failure("AGENT_ACTION_INVALID", "tap needs a label, an identifier, or frame coordinates.");
return { action: { action: "tap", ...selector } as BlackBoxAction, kind: "press", label, risk };
case "fill": if (!selector) throw failure("AGENT_ACTION_INVALID", "fill needs a label or identifier."); return { action: { action: "fill", text: text(input.text, "text", 2000), ...selector } as BlackBoxAction, kind: "fill", label, risk };
case "type": return { action: { action: "type", text: text(input.text, "text", 2000) }, kind: "fill", label: "type", risk: "safe" };
case "fill": if (!selector) throw failure("AGENT_ACTION_INVALID", "fill needs a label or identifier."); return { action: { action: "fill", text: text(input.text, "text", TOOL_LIMITS.text), ...selector } as BlackBoxAction, kind: "fill", label, risk };
case "type": return { action: { action: "type", text: text(input.text, "text", TOOL_LIMITS.text) }, kind: "fill", label: "type", risk: "safe" };
case "scroll": return { action: { action: "scroll", direction: input.direction as "up" }, kind: "swipe", label: `scroll ${String(input.direction)}`, risk: "safe" };
case "wait": return { action: { action: "wait", milliseconds: Number(input.milliseconds) || 1000 }, kind: "other", label: "wait", risk: "safe" };
case "back": return { action: { action: "back" }, kind: "back", label: "back", risk: "safe" };
case "dismiss-keyboard": return { action: { action: "dismiss-keyboard" }, kind: "dismiss", label: "dismiss keyboard", risk: "safe" };
case "open": return { action: { action: "open", url: text(input.url, "url", 2000) }, kind: "deep-link", label: text(input.url, "url", 2000), risk: "unknown" };
case "open": return { action: { action: "open", url: text(input.url, "url", TOOL_LIMITS.url) }, kind: "deep-link", label: text(input.url, "url", TOOL_LIMITS.url), risk: "unknown" };
default: throw failure("AGENT_ACTION_INVALID", `Unknown action ${action}.`);
}
};
@@ -707,8 +734,12 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
const role = roleOf(under);
const label = str(under.label) ?? "";
const what = role === "field" ? "field" : role === "button" || role === "tab" || role === "link" ? (cleanLabel(label) ? role : `icon ${role}`) : /\p{L}/u.test(cleanLabel(label)) ? "item" : "icon";
const row = point.y < 1 / 3 ? "top" : point.y < 2 / 3 ? "" : "bottom";
const column = point.x < 1 / 3 ? "left" : point.x < 2 / 3 ? "" : "right";
// The place is read off the screen's stored screenshot, which an opened Duo's frames are turned upright for: there the
// portrait point (x, y) shows at (1 - y, x). A frame read without an image says nothing, so the screen's screenshot is asked.
const turned = latest!.landscape || map.observations.some(item => item.screenId === currentScreenId && map.artifacts.find(artifact => artifact.id === item.screenshotArtifactId)?.turned);
const shown = turned ? { x: 1 - point.y, y: point.x } : point;
const row = shown.y < 1 / 3 ? "top" : shown.y < 2 / 3 ? "" : "bottom";
const column = shown.x < 1 / 3 ? "left" : shown.x < 2 / 3 ? "" : "right";
const where = [row, column].filter(Boolean).join(" ") || "centre";
return `${what}, ${where}`;
};
@@ -749,10 +780,10 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
const frameId = text(input.frameId, "frameId", 256);
if (!latest?.png && latest?.imageUnavailable?.code === "CAPTURE_SENSITIVE_INPUT") throw failure("CAPTURE_SENSITIVE_INPUT", latest.imageUnavailable.message, true);
if (!latest?.png || latest.frameId !== frameId) throw failure("AGENT_FRAME_STALE", "Observe with an image first; capture must name the latest frame.", true);
const title = text(input.title, "title", 80).trim();
const title = text(input.title, "title", TOOL_LIMITS.title).trim();
if (captures >= options.maxScreens) { stop("budget-screens"); abort.abort(failure("EXPLORATION_BUDGET_EXHAUSTED", "Screen budget exhausted.")); throw abort.signal.reason; }
const { screen } = locate(latest, title);
const state = str(input.state, 80);
const state = str(input.state, TOOL_LIMITS.state);
if (!state && map.observations.some(item => item.screenId === screen.id && item.screenshotArtifactId)) {
if (title) screen.title = title;
return toolResult({ ok: true, duplicate: true, screenId: screen.id, title: screen.title, message: "This screen is already captured (its title is updated); pass state to record a variant." });
@@ -832,9 +863,9 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
await persist();
return toolResult({ ok: !error, outcome, ...(error ? { error } : {}), ...view, ...progress().budget });
}
if (name === "finish") { stop("finished", text(input.summary, "summary", 4000)); abort.abort(new Error("finished")); return toolResult({ ok: true }); }
if (name === "finish") { stop("finished", text(input.summary, "summary", TOOL_LIMITS.summary)); abort.abort(new Error("finished")); return toolResult({ ok: true }); }
if (name === "stop_blocked") {
const reason = text(input.reason, "reason", 1000);
const reason = text(input.reason, "reason", TOOL_LIMITS.reason);
map.gaps.push({ id: `gap_${randomUUID().slice(0, 8)}`, kind: "coverage", status: "open", message: reason, ...(currentScreenId ? { screenId: currentScreenId } : {}), createdAt: timestamp() });
await persist();
stop("blocked", reason); abort.abort(new Error("blocked")); return toolResult({ ok: true });
@@ -856,8 +887,9 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
if (!result) stop(budget ? "budget-time" : "model-error", undefined, budget ? undefined : fatal ?? error);
}
if (result!.stopReason !== "finished" && result!.stopReason !== "blocked") {
const note = result!.stopReason === "model-error" ? undefined : options.budgetNote?.();
map.gaps.push({ id: `gap_${randomUUID().slice(0, 8)}`, kind: result!.stopReason === "model-error" ? "other" : "coverage", status: "open",
message: result!.stopReason === "model-error" ? `The exploration stopped on an error: ${result!.error instanceof Error ? result!.error.message.slice(0, 300) : "unknown"}` : `The exploration stopped at its ${result!.stopReason.replace("budget-", "")} budget with ${map.frontier.filter(item => item.status === "pending").length} controls unexplored.`,
message: result!.stopReason === "model-error" ? `The exploration stopped on an error: ${result!.error instanceof Error ? result!.error.message.slice(0, 300) : "unknown"}` : `The exploration stopped at its ${result!.stopReason.replace("budget-", "")} budget with ${map.frontier.filter(item => item.status === "pending").length} controls unexplored.${note ? ` ${note}` : ""}`,
createdAt: timestamp() });
}
for (const item of map.frontier) if (item.status === "claimed") item.status = "pending";
@@ -870,10 +902,10 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
function nameUncapturedScreens(map: ExploreMap): void {
const titles = new Map(map.screens.map(screen => [screen.id, screen.title]));
for (const screen of map.screens) {
if (!/^Screen \d+$/.test(screen.title)) continue;
if (!isPlaceholderTitle(screen.title)) continue;
const arrival = map.actionAttempts.find(attempt => attempt.targetScreenId === screen.id && attempt.outcome === "transition" && attempt.sourceScreenId !== screen.id);
const from = arrival ? titles.get(arrival.sourceScreenId) : undefined;
if (!arrival || !from || /^Screen \d+$/.test(from)) continue;
if (!arrival || !from || isPlaceholderTitle(from)) continue;
const step = arrival.action.input ? "typing" : arrival.action.label;
screen.title = `${from} → ${step}`.slice(0, 80);
titles.set(screen.id, screen.title);
+300
View File
@@ -0,0 +1,300 @@
import { readFile } from "node:fs/promises";
import { CiError } from "../shared/ci-errors.js";
import type { CiAgentSessionRunRequest, CiAgentSessionRunResult } from "../capture/agent/session.js";
import { SETUP_SELECTOR_MAX, isSetupInput, isSetupSelector } from "../capture/setup-actions.js";
import type { CaptureSetupAction } from "../config/explore-config.js";
import { explorationSecretNames } from "../config/exploration-inputs.js";
import { TOOL_LIMITS, isPlaceholderTitle } from "./explorer.js";
export type ReplayPlatform = "ios" | "android";
type Selector = { label: string; identifier?: never } | { identifier: string; label?: never };
type ReplayAction =
| ({ action: "tap" } & Selector)
| ({ action: "fill"; text: string } & Selector)
| { action: "type"; text: string }
| { action: "scroll"; direction: "up" | "down" | "left" | "right" | "top" | "bottom" }
| { action: "wait"; milliseconds: number }
| { action: "open"; url: string }
| { action: "back" }
| { action: "dismiss-keyboard" };
type ReplayCapture = { capture: string; state?: string; description?: string };
/** `platforms` limits a step to those platforms; an `optional` action that fails is skipped instead of ending the replay. */
export type ReplayStep = (ReplayCapture & { platforms?: ReplayPlatform[] }) | (ReplayAction & { platforms?: ReplayPlatform[]; optional?: boolean });
export type ReplayPlan = { steps: ReplayStep[] };
export type ReplaySession = {
readonly provider: "replay";
readonly model: string;
preflight(): Promise<void>;
run(request: CiAgentSessionRunRequest): Promise<CiAgentSessionRunResult>;
/** After a budget stop, the planned screens the run did not reach, for the explorer's gap. */
budgetNote(): string | undefined;
};
const DIRECTIONS = new Set(["up", "down", "left", "right", "top", "bottom"]);
/**
* The fields each kind of step takes. Anything else is refused: a typo (`platform`, `optinal`) would otherwise do nothing,
* and a field `act` reads another way (a coordinate tap's `frameId`, `x`, `y` and `target`) would change what is tapped.
*/
const FIELDS = new Map<string, readonly string[]>([
["capture", ["capture", "state", "description", "platforms"]],
["tap", ["action", "label", "identifier", "platforms", "optional"]],
["fill", ["action", "label", "identifier", "text", "platforms", "optional"]],
["type", ["action", "text", "platforms", "optional"]],
["scroll", ["action", "direction", "platforms", "optional"]],
["wait", ["action", "milliseconds", "platforms", "optional"]],
["open", ["action", "url", "platforms", "optional"]],
["back", ["action", "platforms", "optional"]],
["dismiss-keyboard", ["action", "platforms", "optional"]],
]);
const ACTIONS = [...FIELDS.keys()].filter(kind => kind !== "capture");
function invalid(message: string): CiError {
return new CiError({ code: "REPLAY_PLAN_INVALID", stage: "exploration", message, retryable: false,
nextAction: "Fix the replay plan: a JSON object with a steps array of actions ({\"action\":\"tap\",\"label\":\"Settings\"}) and captures ({\"capture\":\"Settings\"})." });
}
const nonEmpty = (value: unknown, max: number): value is string => typeof value === "string" && value.trim().length > 0 && value.length <= max;
function parseStep(step: unknown, at: string): ReplayStep {
if (!step || typeof step !== "object" || Array.isArray(step)) throw invalid(`${at} is not an object.`);
const entry = step as Record<string, unknown>;
const kind = "capture" in entry ? "capture" : typeof entry.action === "string" && FIELDS.has(entry.action) ? entry.action : undefined;
if (!kind) throw invalid(`${at} needs a capture title or an action (${ACTIONS.join(", ")}).`);
const fields = FIELDS.get(kind)!;
const unknown = Object.keys(entry).find(key => !fields.includes(key));
if (unknown !== undefined) throw invalid(`${at} (${kind}) has a field ${JSON.stringify(unknown.slice(0, 40))} it does not take; a ${kind} step takes ${fields.join(", ")}.`);
if (entry.platforms !== undefined && (!Array.isArray(entry.platforms) || !entry.platforms.length || entry.platforms.some(item => item !== "ios" && item !== "android"))) {
throw invalid(`${at}: platforms must list ios and/or android.`);
}
if (entry.optional !== undefined && typeof entry.optional !== "boolean") throw invalid(`${at}: optional must be true or false.`);
const extra = { ...(entry.platforms ? { platforms: [...entry.platforms] as ReplayPlatform[] } : {}), ...(entry.optional !== undefined ? { optional: entry.optional as boolean } : {}) };
if (kind === "capture") {
if (!nonEmpty(entry.capture, TOOL_LIMITS.title)) throw invalid(`${at}: capture must be a screen title of at most ${TOOL_LIMITS.title} characters.`);
if (entry.state !== undefined && !nonEmpty(entry.state, TOOL_LIMITS.state)) throw invalid(`${at}: state must be a string of at most ${TOOL_LIMITS.state} characters.`);
if (entry.description !== undefined && !nonEmpty(entry.description, TOOL_LIMITS.description)) throw invalid(`${at}: description must be a string of at most ${TOOL_LIMITS.description} characters.`);
return { capture: entry.capture.trim(), ...(entry.state !== undefined ? { state: entry.state as string } : {}),
...(entry.description !== undefined ? { description: entry.description as string } : {}), ...extra };
}
// The device selects by the setup executor's rules, which are stricter than what `act` itself accepts.
const selector = (): Selector => {
const keys = (["label", "identifier"] as const).filter(key => entry[key] !== undefined);
if (keys.length !== 1) throw invalid(`${at}: ${kind} needs exactly one of label or identifier.`);
const key = keys[0]!, value = entry[key];
if (!isSetupSelector(value) || value.length > TOOL_LIMITS.selector) throw invalid(`${at}: ${key} must be 1–${SETUP_SELECTOR_MAX} characters without control characters.`);
return key === "label" ? { label: value } : { identifier: value };
};
const input = (): string => {
if (!nonEmpty(entry.text, TOOL_LIMITS.text) || !isSetupInput(entry.text)) throw invalid(`${at}: ${kind} needs text: one line of at most ${TOOL_LIMITS.text} characters.`);
try { explorationSecretNames(entry.text); }
catch (error) { throw invalid(`${at}: ${error instanceof Error ? error.message : String(error)} Write each input as {{secret:NAME}} with an uppercase environment variable name that is not a provider, Expo, or CI credential.`); }
return entry.text;
};
switch (kind) {
case "tap": return { action: "tap", ...selector(), ...extra };
case "fill": return { action: "fill", ...selector(), text: input(), ...extra };
case "type": return { action: "type", text: input(), ...extra };
case "scroll":
if (typeof entry.direction !== "string" || !DIRECTIONS.has(entry.direction)) throw invalid(`${at}: scroll needs a direction (${[...DIRECTIONS].join(", ")}).`);
return { action: "scroll", direction: entry.direction as "up", ...extra };
case "wait":
if (!(Number.isInteger(entry.milliseconds) && Number(entry.milliseconds) >= 1 && Number(entry.milliseconds) <= TOOL_LIMITS.waitMs)) {
throw invalid(`${at}: wait needs milliseconds between 1 and ${TOOL_LIMITS.waitMs}.`);
}
return { action: "wait", milliseconds: entry.milliseconds as number, ...extra };
case "open":
if (!nonEmpty(entry.url, TOOL_LIMITS.url)) throw invalid(`${at}: open needs a url of at most ${TOOL_LIMITS.url} characters.`);
return { action: "open", url: entry.url, ...extra };
default: return { action: kind as "back" | "dismiss-keyboard", ...extra };
}
}
/** Checks a plan and keeps only the fields each step takes, so what reaches `act` is what the plan documents. */
export function parseReplayPlan(value: unknown): ReplayPlan {
if (!value || typeof value !== "object" || !Array.isArray((value as { steps?: unknown }).steps)) throw invalid("The replay plan has no steps array.");
const unknown = Object.keys(value).find(key => key !== "steps");
if (unknown !== undefined) throw invalid(`The replay plan has a field ${JSON.stringify(unknown.slice(0, 40))} it does not take; it takes only steps.`);
const steps = (value as { steps: unknown[] }).steps;
if (!steps.length) throw invalid("The replay plan has no steps.");
return { steps: steps.map((step, index) => parseStep(step, `Step ${index + 1}`)) };
}
export async function readReplayPlan(path: string): Promise<ReplayPlan> {
let text: string;
try { text = await readFile(path, "utf8"); }
catch (error) { throw invalid(`The replay plan ${path} could not be read: ${error instanceof Error ? error.message : String(error)}`); }
let value: unknown;
try { value = JSON.parse(text); }
catch (error) { throw invalid(`The replay plan ${path} is not JSON: ${error instanceof Error ? error.message : String(error)}`); }
return parseReplayPlan(value);
}
/** The plan's fill and type steps, whose {{secret:NAME}} inputs must be set before any device starts, as the hints' are. */
export function replaySetup(plan: ReplayPlan): CaptureSetupAction[] {
return plan.steps.flatMap(step => "action" in step && (step.action === "fill" || step.action === "type") ? [step as CaptureSetupAction] : []);
}
const stepsFor = (plan: ReplayPlan, platform: ReplayPlatform) => plan.steps.map((step, index) => ({ step, number: index + 1 }))
.filter(({ step }) => !step.platforms || step.platforms.includes(platform));
/**
* Refuses, before any device starts, a plan with nothing to do on this platform or one the budgets cannot hold: a run that
* would stop at its budget partway through the plan is better raised or shortened up front.
*/
export function assertReplayFits(plan: ReplayPlan, platform: ReplayPlatform, limits: { maxScreens: number; maxActions: number; maxTurns: number }): void {
const steps = stepsFor(plan, platform);
if (!steps.length) {
throw new CiError({ code: "REPLAY_PLAN_EMPTY", stage: "exploration", message: `No step of the replay plan runs on ${platform}: each one is limited to ${platform === "ios" ? "android" : "ios"} by its platforms.`,
nextAction: `Add steps for ${platform} to the plan, or run it on the platform it was written for.` });
}
const captures = steps.filter(({ step }) => "capture" in step).length;
const actions = steps.filter(({ step }) => "action" in step && step.action !== "wait").length;
// A screen counts against the budget when it is first photographed, on arrival, and a capture is refused once the budget
// is used up, so the last capture needs one screen to spare.
const over = [
...(captures >= limits.maxScreens ? [{ text: `${captures} captures, which need a screen budget of at least ${captures + 1} (it is ${limits.maxScreens})`, flag: `--max-screens ${captures + 1}` }] : []),
...(actions > limits.maxActions ? [{ text: `${actions} actions, more than the action budget of ${limits.maxActions}`, flag: `--max-actions ${actions}` }] : []),
];
if (over.length) {
throw new CiError({ code: "REPLAY_PLAN_OVER_BUDGET", stage: "exploration", message: `On ${platform} the replay plan takes ${over.map(item => item.text).join(" and ")}.`,
nextAction: `Pass ${over.map(item => item.flag).join(" ")} (or more: each screen the plan reaches without capturing it also counts), or shorten the plan.` });
}
// Each capture is an observe and a capture call, each action one act call, and the run ends with get_progress and finish.
const turns = steps.length + captures + 2;
if (turns > limits.maxTurns) {
throw new CiError({ code: "REPLAY_PLAN_OVER_BUDGET", stage: "exploration", message: `On ${platform} the replay plan takes ${turns} tool calls, more than the ${limits.maxTurns} that the screen and action budgets allow.`,
nextAction: "Raise --max-actions, or use fewer wait steps." });
}
}
function describeAction(step: ReplayAction): string {
if ("label" in step && step.label) return `${step.action} "${step.label}"`;
if ("identifier" in step && step.identifier) return `${step.action} #${step.identifier}`;
if (step.action === "scroll") return `scroll ${step.direction}`;
return step.action;
}
/** Only the fields `act` documents for that action, whatever else the step object carries. */
function actInput(step: ReplayAction): Record<string, unknown> {
const selector = "label" in step && step.label !== undefined ? { label: step.label } : "identifier" in step && step.identifier !== undefined ? { identifier: step.identifier } : {};
switch (step.action) {
case "tap": return { action: "tap", ...selector };
case "fill": return { action: "fill", ...selector, text: step.text };
case "type": return { action: "type", text: step.text };
case "scroll": return { action: "scroll", direction: step.direction };
case "wait": return { action: "wait", milliseconds: step.milliseconds };
case "open": return { action: "open", url: step.url };
default: return { action: step.action };
}
}
/** As many names as fit in `room` characters, then "and N more"; a count of them when not even the first fits. */
function listed(names: readonly string[], room: number, noun: string): string {
let text = "";
for (const [index, name] of names.entries()) {
const next = text ? `${text}, ${name}` : name;
const left = names.length - index - 1;
if (next.length + (left ? ` and ${left} more`.length : 0) > room) return text ? `${text} and ${names.length - index} more` : `${names.length} ${noun}`;
text = next;
}
return text;
}
const sameTitle = (left: string, right: string): boolean => left.toLowerCase() === right.toLowerCase();
/**
* Runs a fixed plan through the explorer's own tools in place of a model: the same observe, act, and capture calls,
* so the map and report are those of a model run, but every run takes the same steps. A step that fails does not
* fail the run: the report keeps what was captured and names the screens that were not.
*/
export function createReplaySession(plan: ReplayPlan, platform: ReplayPlatform): ReplaySession {
const steps = stepsFor(plan, platform);
const title = (step: ReplayCapture) => step.state ? `${step.capture} (${step.state})` : step.capture;
const missed: string[] = [];
let position = 0;
/** What the plan meant to capture and has not: the screens still ahead first, then the ones earlier steps missed. */
const notCaptured = () => [...steps.slice(position).flatMap(({ step }) => "capture" in step ? [title(step)] : []), ...missed];
return {
provider: "replay",
model: "replay plan",
async preflight() {},
budgetNote() {
const names = notCaptured();
return names.length ? `Not captured: ${listed(names, TOOL_LIMITS.reason, "planned screens")}.` : undefined;
},
async run(request) {
const signal = request.signal ?? new AbortController().signal;
const call = async (name: string, input: Record<string, unknown>) => {
const result = await request.callTool(name, input, signal);
const text = result.content.find(block => block.type === "text");
const value = text && "text" in text ? JSON.parse(text.text) as Record<string, unknown> : {};
return { failed: Boolean(result.isError) || value.ok === false, value };
};
const why = (value: Record<string, unknown>) => String(value.error ?? value.reason ?? value.message ?? "it failed");
// A refused finish or stop_blocked would otherwise end the session with no result, which the explorer reports as a turn budget stop.
const end = async (name: "finish" | "stop_blocked", input: Record<string, unknown>) => {
const ended = await call(name, input);
if (!ended.failed) return;
if (name === "finish") return end("stop_blocked", { reason: `The replay could not finish (${why(ended.value)}). ${String(input.summary)}`.slice(0, TOOL_LIMITS.reason) });
throw new CiError({ code: "REPLAY_STOP_REFUSED", stage: "exploration", message: `The explorer refused the replay's ${name}: ${why(ended.value)}`.slice(0, 600),
nextAction: "Report this with the replay plan; the screens captured so far are in the map." });
};
/** Screen id to the title the plan captured it under. */
const planned = new Map<string, string>();
const skipped: string[] = [];
missed.length = 0;
try {
for (const [index, { step, number }] of steps.entries()) {
position = index;
if ("capture" in step) {
const seen = await call("observe", { includeImage: true });
const frameId = typeof seen.value.frameId === "string" ? seen.value.frameId : undefined;
const known = typeof seen.value.screenId === "string" && !step.state ? planned.get(seen.value.screenId) : undefined;
// The plan already captured this screen: a second capture would only rename it, so it is not made.
if (known !== undefined) {
if (!sameTitle(known, step.capture)) missed.push(`${title(step)} (looked the same as ${known})`);
continue;
}
const captured = frameId && !seen.failed ? await call("capture", { frameId, title: step.capture,
...(step.state ? { state: step.state } : {}), ...(step.description ? { description: step.description } : {}) }) : undefined;
const screenId = typeof captured?.value.screenId === "string" ? captured.value.screenId : undefined;
if (!captured || captured.failed || !screenId) missed.push(title(step));
else if (!planned.has(screenId)) planned.set(screenId, step.capture);
continue;
}
const acted = await call("act", actInput(step));
if (!acted.failed) continue;
if (step.optional) { skipped.push(`step ${number} (${describeAction(step)})`); continue; }
const head = `Replay step ${number} (${describeAction(step)}) failed: ${why(acted.value)}`.slice(0, 600);
const names = notCaptured();
const lead = ". Not captured: ";
await end("stop_blocked", { reason: head + (names.length ? `${lead}${listed(names, TOOL_LIMITS.reason - head.length - lead.length, "planned screens")}` : "") });
return { inputTokens: 0, outputTokens: 0 };
}
position = steps.length;
// The map as it stands: screens merged into others are gone, and each keeps its latest title. One reached but never
// captured is still "Screen N" here and is named after the path to it only once the run ends, so it is counted.
const progress = await call("get_progress", {});
const screens = Array.isArray(progress.value.captured) ? progress.value.captured as Array<{ id?: unknown; title?: unknown }> : [];
const unplanned = screens.filter(screen => typeof screen.id === "string" && !planned.has(screen.id)).map(screen => typeof screen.title === "string" ? screen.title : "");
const named = unplanned.filter(name => name && !isPlaceholderTitle(name));
const unnamed = unplanned.length - named.length;
const reached = [...named, ...(unnamed ? [`${unnamed} unnamed screen${unnamed === 1 ? "" : "s"}`] : [])];
const sections = ([["Not captured", missed, "planned screens"], ["Optional steps skipped because they failed", skipped, "steps"],
["Also reached, not in the plan", reached, "screens"]] as const).filter(([, names]) => names.length);
let summary = `Replayed ${steps.length} planned steps on ${platform}.${missed.length ? "" : " Every planned screen was captured."}`;
for (const [index, [label, names, noun]] of sections.entries()) {
// Each later section keeps room for its label and a count, so a long list of misses does not crowd out the rest.
const reserve = sections.slice(index + 1).reduce((sum, [later]) => sum + later.length + 32, 0);
const lead = ` ${label}: `;
summary += `${lead}${listed(names, TOOL_LIMITS.summary - summary.length - lead.length - 1 - reserve, noun)}.`;
}
await end("finish", { summary });
} catch (error) {
if (signal.aborted) return { inputTokens: 0, outputTokens: 0 };
throw error;
}
return { inputTokens: 0, outputTokens: 0 };
},
};
}
+83 -10
View File
@@ -5,6 +5,7 @@ import { join } from "node:path";
import { createCiAgentSession, type CiAgentSession, type CiAgentSessionConfig } from "../capture/agent/session.js";
import { createLocalCiAgentSession } from "../capture/agent/session-local.js";
import type { CiAgentProviderName } from "../capture/agent/provider.js";
import { assertReplayFits, createReplaySession, parseReplayPlan, replaySetup, type ReplayPlan } from "./replay.js";
import { CONTROLLER_SESSION, baseEnvironment, isRecord } from "../capture/common.js";
import { controllerData, defaultRunner, openedSimulatorDevice } from "../capture/controller.js";
import type { CommandRequest, CommandResult } from "../capture/run.js";
@@ -16,8 +17,10 @@ import { agentDeviceLauncher, ensureArgentToolServer, ensureController } from ".
import { agentDeviceClaims, approveLinkScheme, createLocalSimulator, deleteLocalEmulator, deleteLocalSimulator, hostEmulatorAbi, installOnLocalEmulator, installOnLocalSimulator, startLocalArgentServer,
startLocalEmulator, stopLocalAgentDevice, stopLocalArgentServer, waitForLocalEmulator, waitForLocalSimulator, type LocalArgentServer } from "../capture/local-simulator.js";
import { openLocalCaptureRecord, sweepLocalCaptures } from "../capture/local-registry.js";
import { screenSpace } from "../capture/route-capture.js";
import { approveSessionUrlSchemes, simulatorControlActions, type EasCommand } from "../capture/simulator-control.js";
import { DEFAULT_ANDROID_DEVICE } from "../config/explore-config.js";
import { DUO_PANELS } from "../contracts/capture.js";
import { createExplorationInputs } from "../config/exploration-inputs.js";
import { ExploreStore, getCoverage } from "../map/store.js";
import { whoamiAccounts } from "../report/preview-deploy.js";
@@ -57,6 +60,8 @@ export type BinaryExploreOptions = {
simulator?: "eas" | "local";
environment?: NodeJS.ProcessEnv;
runner?: typeof defaultRunner;
/** Fixed steps to take instead of a model's; no model access is needed. */
replay?: ReplayPlan;
createSession?: (config: CiAgentSessionConfig, options: { env: NodeJS.ProcessEnv; auth: ModelAuth }) => CiAgentSession;
progress?: (message: string) => void;
now?: () => number;
@@ -68,7 +73,7 @@ export type BinaryExploreResult = {
archive: Omit<InspectedArchive, "path"> & { path: string };
session: { id: string; platform: Platform; device: string; owner: string; project: string };
runtime: ActiveSimulatorRuntime;
model: { provider: CiAgentProviderName; model: string; auth: "local-login" | "api-key" };
model: { provider: CiAgentProviderName | "replay"; model: string; auth: "local-login" | "api-key" | "none" };
exploration: BlackBoxExplorerResult;
durationMs: number;
};
@@ -162,8 +167,8 @@ export function resolveSessionApp(runner: typeof defaultRunner, connection: Simu
* App-test inputs: each {{secret:NAME}} in the hints names an environment variable whose value Explore types for the
* model. Read before any session starts, so a missing one costs nothing.
*/
export function hintInputs(hints: string | undefined, environment: NodeJS.ProcessEnv): ReturnType<typeof createExplorationInputs> {
try { return createExplorationInputs({ instructions: hints ?? "", environment }); }
export function hintInputs(hints: string | undefined, environment: NodeJS.ProcessEnv, replay?: ReplayPlan): ReturnType<typeof createExplorationInputs> {
try { return createExplorationInputs({ instructions: hints ?? "", ...(replay ? { setup: replaySetup(replay) } : {}), environment }); }
catch (error) {
if (!(error instanceof CiError) || !error.code.startsWith("EXPLORATION_SECRET_")) throw error;
const name = typeof error.details?.secretName === "string" ? error.details.secretName : "NAME";
@@ -194,6 +199,36 @@ function modelSession(provider: CiAgentProviderName, auth: ModelAuth, options: M
return { session, provider, model: config.model, auth };
}
/** The turns the model gets: three for each action and two for each screen, plus a few to look around and finish. */
const turnBudget = (limits: { maxScreens: number; maxActions: number }): number => limits.maxActions * 3 + limits.maxScreens * 2 + 20;
/**
* A replay plan is checked here as well as by the CLI, for callers that pass one directly. It takes the model's place, so
* the options that only steer a model would do nothing; the plan's own fill and type steps name the secrets it types.
*/
function replayPlan(options: Pick<BinaryExploreOptions, "replay" | "provider" | "model" | "hints">): ReplayPlan | undefined {
if (!options.replay) return undefined;
const ignored = (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
if (ignored.length) {
throw binaryError("REPLAY_OPTIONS_IGNORED", `--replay takes the model's place, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
`Drop ${ignored.map(name => `--${name}`).join(" and ")}. A {{secret:NAME}} in the plan's fill and type steps is read from the environment, as one in --hints is.`);
}
const plan = parseReplayPlan(options.replay);
for (const [index, step] of plan.steps.entries()) {
if (!("action" in step) || step.action !== "open") continue;
try { assertAppUrl(step.url); }
catch (error) { throw new CiError({ code: "REPLAY_PLAN_INVALID", stage: "exploration", message: `Step ${index + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the replay plan." }); }
}
return plan;
}
/** A replay plan takes the model's place: no model access is checked or used. */
function replayModel(plan: ReplayPlan, platform: Platform, limits: { maxScreens: number; maxActions: number }) {
assertReplayFits(plan, platform, { ...limits, maxTurns: turnBudget(limits) });
const session = createReplaySession(plan, platform);
return { session, budgetNote: session.budgetNote, provider: "replay" as const, model: "replay plan", auth: "none" as const };
}
/** Local `claude`/`codex` login unless an API key is present; preflight proves access before a simulator is paid for. */
export function selectModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
const provider: CiAgentProviderName = options.provider ?? (environment.OPENAI_API_KEY?.trim() && !environment.ANTHROPIC_API_KEY?.trim() ? "openai" : "anthropic");
@@ -260,7 +295,7 @@ export type ConnectedSessionExploration = {
limits: { maxDurationMinutes: number; maxScreens: number; maxActions: number };
store: ExploreStore;
map: ExploreMap;
model: { session: CiAgentSession };
model: { session: Pick<CiAgentSession, "run">; budgetNote?: () => string | undefined };
/** `eas simulator:get --json` for the attached session; descriptive (OS version) only. */
sessionJson: () => string;
/** agent-device session name on the daemon; explore:session reuses the caller's session so the device is not "in use" by another. */
@@ -374,6 +409,20 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
catch { return undefined; }
};
const foregroundElsewhere = (): boolean => { const state = foregroundState(); return state ? leftForeground(state, bundleId) : false; };
let unmatchedPanel = false;
/** Whether the frame is an opened Duo's, to be stored turned upright; the run then records the landscape it shows. */
const landscapeOnPortrait = (nodes: readonly Record<string, unknown>[], png: Buffer): boolean => {
const width = png.readUInt32BE(16), height = png.readUInt32BE(20);
const turn = duoScreenshotTurn({ platform, deviceModel: runtime.deviceModel, nodes, width, height });
if (turn === "unmatched" && !unmatchedPanel) {
unmatchedPanel = true;
const root = nodes.find(node => node.type === "Application")?.rect as { width?: unknown; height?: unknown } | undefined;
say(`an iPhone Duo frame (${width}×${height} px, root ${root ? `${String(root.width)}×${String(root.height)} pt` : "missing"}) shows neither of its panels; its screenshot is stored as taken, which may be sideways`);
}
if (turn !== "turn") return false;
if (runtime.orientation !== "landscape") { runtime.orientation = "landscape"; if (map.run.environment) map.run.environment.orientation = "landscape"; }
return true;
};
const observe = async (includeImage: boolean): Promise<BlackBoxFrame> => {
let nodes: Record<string, unknown>[];
let leftApp = false;
@@ -411,7 +460,7 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
assertDecodedPng(png, "Exploration screenshot");
if (png.length > 10 * 1024 * 1024) return { nodes, capturedAt, imageUnavailable: { code: "SCREENSHOT_TOO_LARGE", message: "The screenshot exceeded 10 MB." } };
if (solidColour(png)) return { nodes, capturedAt, imageUnavailable: { code: "SCREENSHOT_BLANK", message: BLANK_SCREENSHOT_MESSAGE }, ...(leftApp ? { leftApp } : {}) };
return { nodes, capturedAt, frameId: randomUUID(), png, imageSha256: createHash("sha256").update(png).digest("hex"), width: png.readUInt32BE(16), height: png.readUInt32BE(20), ...(leftApp ? { leftApp } : {}) };
return { nodes, capturedAt, frameId: randomUUID(), png, imageSha256: createHash("sha256").update(png).digest("hex"), width: png.readUInt32BE(16), height: png.readUInt32BE(20), ...(landscapeOnPortrait(nodes, png) ? { landscape: true } : {}), ...(leftApp ? { leftApp } : {}) };
} finally { await rm(path, { force: true }); }
};
const setupOptions = { command: async (args: string[]) => { const result = controller(guardPositionals([...args, "--json"]), 120_000); return controllerData(result.stdout, args[0] ?? "command"); },
@@ -444,7 +493,8 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
}
const exploration = blockedAtLaunch ?? await runBlackBoxExploration({ session: model.session, store, map, app: { name: app.name, bundleId, platform, ...(app.version ? { version: app.version } : {}) },
inputs, ...(hints ? { hints: hints } : {}), ...(launchUrl ? { launchUrl: launchUrl } : {}), ...(allowDestructive ? { allowDestructive: allowDestructive } : {}), ...(ctx.appIcon ? { appIcon: ctx.appIcon } : {}),
deadlineMs, maxScreens: limits.maxScreens, maxActions: limits.maxActions, maxTurns: limits.maxActions * 3 + limits.maxScreens * 2 + 20, now, observe, progress: say,
...(model.budgetNote ? { budgetNote: model.budgetNote } : {}),
deadlineMs, maxScreens: limits.maxScreens, maxActions: limits.maxActions, maxTurns: turnBudget(limits), now, observe, progress: say,
act: async action => {
const perform = async () => {
if (action.action === "tap" && "frameId" in action) {
@@ -508,7 +558,8 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
const environment = options.environment ?? process.env;
const runner = options.runner ?? defaultRunner;
const limits = validatedLimits(options);
const inputs = hintInputs(options.hints, environment);
const replay = replayPlan(options);
const inputs = hintInputs(options.hints, environment, replay);
if (options.bundleId !== undefined && !appIdPattern.test(options.bundleId)) throw binaryError("BUNDLE_ID_INVALID", `${options.bundleId} is not a bundle or package id.`, "Pass --bundle-id with the app's bundle identifier (iOS) or package name (Android).");
let connection: SimulatorConnection;
try { connection = await resolveAttachedSimulatorSession(environment, options.envFile, options.session); }
@@ -572,7 +623,7 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
map.run.environment = { deviceModel: device };
session = { id: connection.sessionId, platform, device, name: info.name, ...(info.deviceRunSessionUrl ? { url: info.deviceRunSessionUrl } : {}) };
const model = await preflightModelSession(options, environment, say);
const model = replay ? replayModel(replay, platform, limits) : await preflightModelSession(options, environment, say);
const outputDirectory = await realpath(await mkdir(options.output, { recursive: true }).then(() => options.output));
const store = new ExploreStore(outputDirectory);
if (await store.exists()) throw binaryError("OUTPUT_NOT_EMPTY", `${outputDirectory} already holds an Explore map.`, "Pass an empty --output directory.");
@@ -665,7 +716,8 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
const environment = options.environment ?? process.env;
const runner = options.runner ?? defaultRunner;
const limits = validatedLimits(options);
const inputs = hintInputs(options.hints, environment);
const replay = replayPlan(options);
const inputs = hintInputs(options.hints, environment, replay);
const local = options.simulator === "local";
// A local emulator runs this Mac's ABI (arm64-v8a on Apple silicon); EAS's runs x86_64.
const emulatorAbi = local ? hostEmulatorAbi() : "x86_64";
@@ -682,7 +734,7 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
say(`${archive.appName} ${archive.version ?? ""} (${bundleId}, ${platform}, ${(archive.bytes / 1048576).toFixed(0)} MB, sha256 ${archive.sha256.slice(0, 12)})`);
if (archive.missingLibs?.length && emulatorAbi === "x86_64") say(`warning: lib/x86_64 lacks ${archive.missingLibs.slice(0, 4).join(", ")}${archive.missingLibs.length > 4 ? ` and ${archive.missingLibs.length - 4} more` : ""} that another ABI ships; the app may stop with UnsatisfiedLinkError on the emulator`);
const model = await preflightModelSession(options, environment, say);
const model = replay ? replayModel(replay, platform, limits) : await preflightModelSession(options, environment, say);
const outputDirectory = await realpath(await mkdir(options.output, { recursive: true }).then(() => options.output));
const store = new ExploreStore(outputDirectory);
@@ -871,6 +923,27 @@ export function guardPositionals(args: readonly string[]): string[] {
const KEYBOARD_RETURN_KEY = /^(return|next|done|go|continue|join)$/i;
/**
* How a frame's screenshot is stored. An opened (or half-open) iPhone Duo lights its inner panel, whose native axes are
* portrait, and shows a landscape UI on it: the tree and taps stay on the portrait axes (a 669×951 pt root, the transpose
* of `DUO_PANELS.inner`), and a screenshot on those axes is sideways to a person, so it is turned ("turn"). Local
* agent-device runs returned exactly that (2007×2853 px). Capture's Argent screenshots on EAS, taken with Argent's
* LandscapeRight rotation, came back upright (2853×2007); an image that is already the root's transpose is left as it is.
* The root is the full-screen Application node capture takes its coordinate space from, matched to within a pixel and a
* point. A Duo frame on neither panel is "unmatched": it is stored as taken, and the caller says so. An empty tree, read
* mid-draw, has nothing to judge.
*/
export function duoScreenshotTurn(frame: { platform: Platform; deviceModel: string; nodes: readonly Record<string, unknown>[]; width: number; height: number }): "turn" | "keep" | "unmatched" {
if (frame.platform !== "ios" || !/\bduo\b/i.test(frame.deviceModel) || !frame.nodes.length) return "keep";
const space = screenSpace(frame.nodes, frame, 1);
if (!space) return "unmatched";
const near = (value: number, points: number) => Math.abs(value - points) < 1;
const portraitAxes = near(space.width, DUO_PANELS.inner.height) && near(space.height, DUO_PANELS.inner.width);
if (portraitAxes && !space.transposed && frame.width < frame.height) return "turn";
const panel = Object.values(DUO_PANELS).some(({ width, height }) => (near(space.width, width) && near(space.height, height)) || (near(space.width, height) && near(space.height, width)));
return panel ? "keep" : "unmatched";
}
/** A node that is not the app's UI: another package drew it (Android's system UI over the app), or it is Explore's own route marker, which an app
* instrumented for PR runs renders as a 2x2 hidden text. */
export function foreignNode(node: Record<string, unknown>, bundleId: string): boolean {
+35 -5
View File
@@ -5,6 +5,7 @@
* injected, so the translation is tested against recorded Argent output.
*/
import { CONTROLLER_SESSION } from "./common.js";
import { INPUT_METHOD_PACKAGES, KEYBOARD_WINDOW_IDENTIFIERS } from "./keyboard.js";
import type { DecodedFrame } from "../shared/png-evidence.js";
export type ArgentPlatform = "ios" | "android";
@@ -88,7 +89,7 @@ export function parseCommand(argv: readonly string[]): { positional: string[]; f
}
type ArgentNode = { role?: string; type?: string; label?: string; identifier?: string; value?: string; contentDescription?: string; disabled?: boolean;
frame?: Rect; traits?: string[]; hidden?: boolean; selected?: boolean; password?: boolean; bundleId?: string; children?: ArgentNode[] };
frame?: Rect; traits?: string[]; hidden?: boolean; selected?: boolean; focused?: boolean; password?: boolean; bundleId?: string; children?: ArgentNode[] };
export type ArgentTree = { schemaVersion?: number; source?: string; screen?: { width: number; height: number }; roots?: ArgentNode[];
truncated?: boolean; alertVisible?: boolean; keyboardVisible?: boolean; foregroundApp?: string };
@@ -137,7 +138,11 @@ export function snapshotFromTree(tree: ArgentTree, platform: ArgentPlatform, gen
if (root || !node.frame || ![node.frame.x, node.frame.y, node.frame.width, node.frame.height].every(finite)) return clip;
return clip && intersect(node.frame, clip);
};
const visit = (node: ArgentNode, depth: number, parentIndex: number | undefined, appRoot: boolean, clip: Rect | undefined, root: boolean) => {
const keyboard = platform === "ios" ? keyboardWindows(roots) : new Set<ArgentNode>();
/** `owner` marks every node of a window that is the system keyboard's, as agent-device marks an input method's nodes. */
type Owner = { bundleId?: string; systemKeyboard?: true };
const visit = (node: ArgentNode, depth: number, parentIndex: number | undefined, appRoot: boolean, clip: Rect | undefined, root: boolean, owner: Owner) => {
if (keyboard.has(node)) owner = { ...owner, systemKeyboard: true };
const frame = node.frame;
const inner = clipOf(node, clip, root);
const rect = frame && [frame.x, frame.y, frame.width, frame.height].every(finite) ? toPoints(frame) : undefined;
@@ -159,19 +164,42 @@ export function snapshotFromTree(tree: ArgentTree, platform: ArgentPlatform, gen
rect: appRoot ? { x: 0, y: 0, width: screen.width, height: screen.height } : rect,
...(label ? { label } : {}), ...(node.identifier ? { identifier: node.identifier } : {}),
...(node.value !== undefined ? { value: node.value } : {}), enabled, hittable: enabled && visible && !appRoot,
...(visible ? {} : { visible: false }), ...(node.selected || traits.includes("selected") ? { selected: true } : {}) });
...(visible ? {} : { visible: false }), ...(node.selected || traits.includes("selected") ? { selected: true } : {}),
// The field with the keyboard focus, as agent-device marks it, so capture can wait for the keyboard it raises.
...(node.focused === true || traits.includes("isEditing") ? { focused: true } : {}),
...owner });
refs[ref] = { rect: appRoot ? { x: 0, y: 0, width: screen.width, height: screen.height } : rect, type,
...(node.value !== undefined ? { value: node.value } : {}), ...(type === "SecureTextField" ? { secure: true } : {}) };
}
for (const child of node.children ?? []) visit(child, depth + 1, index, false, inner, false);
for (const child of node.children ?? []) visit(child, depth + 1, index, false, inner, false, owner);
};
for (const root of roots) {
const isAppRoot = platform === "ios" ? root.type === "Application" || nodes.length === 0 : root.bundleId === app;
visit(root, 0, undefined, isAppRoot, { x: 0, y: 0, width: 1, height: 1 }, true);
// An Android input method's window names its package on each node, as agent-device's do, which marks the keyboard.
const inputMethod = platform === "android" && root.bundleId && INPUT_METHOD_PACKAGES.has(root.bundleId) ? root.bundleId : undefined;
visit(root, 0, undefined, isAppRoot, { x: 0, y: 0, width: 1, height: 1 }, true, inputMethod ? { bundleId: inputMethod } : {});
}
return { nodes, refs, screen: { width: screen.width, height: screen.height } };
}
/**
* The windows iOS draws the software keyboard in. Argent lists each window as a child of its root, the app's first. A
* later window is the keyboard's when it holds XCTest's `Keyboard` element, which no app can give itself; an identifier
* never decides it alone, since an app can set any identifier on its own views (a second window of the app's with a
* `testID="inputView"` stays the app's). While such a window is up, UIKit also hosts the keyboard's input view
* placeholder in a window of its own: a later window whose only named node is an `inputView` belongs to it too.
*/
function keyboardWindows(roots: readonly ArgentNode[]): Set<ArgentNode> {
const holds = (node: ArgentNode, test: (node: ArgentNode) => boolean): boolean => test(node) || (node.children ?? []).some(child => holds(child, test));
const later = roots.flatMap(root => (root.children ?? []).slice(1));
const keyboard = later.filter(window => holds(window, node => node.type === "Keyboard"));
if (!keyboard.length) return new Set();
const named = (node: ArgentNode) => Boolean(node.label?.trim() || node.contentDescription?.trim() || node.value !== undefined
|| (node.identifier && !KEYBOARD_WINDOW_IDENTIFIERS.has(node.identifier)));
const placeholders = later.filter(window => !keyboard.includes(window) && holds(window, node => node.identifier === "inputView") && !holds(window, named));
return new Set([...keyboard, ...placeholders]);
}
function intersect(a: Rect, b: Rect): Rect | undefined {
const x = Math.max(a.x, b.x), y = Math.max(a.y, b.y);
const right = Math.min(a.x + a.width, b.x + b.width), bottom = Math.min(a.y + a.height, b.y + b.height);
@@ -438,7 +466,9 @@ export async function runBridgeCommand(argv: readonly string[], context: ArgentB
converted.nodes.push({ depth: 1, index: converted.nodes.length, parentIndex: 0, type: "Alert", ref: `e${converted.nodes.length + 1}`,
rect: { x: 0, y: 0, width: converted.screen.width, height: converted.screen.height }, label: "System alert", enabled: true, hittable: false });
}
// Whether a keyboard shows, which Argent's Android tree does not hold: the input method's window is not in it.
return success({ nodes: converted.nodes, truncated: current.truncated === true, refsGeneration: state.generation,
...(typeof current.keyboardVisible === "boolean" ? { keyboardVisible: current.keyboardVisible } : {}),
...(current.foregroundApp ? { appBundleId: current.foregroundApp, appName: current.foregroundApp } : {}),
snapshotQuality: { state: "healthy", backend: "argent" } });
}
+8 -4
View File
@@ -5,6 +5,7 @@ import type {
ElementRole,
InputElement,
} from "../types.js";
import { INPUT_METHOD_PACKAGES, keyboardNodes } from "./keyboard.js";
export type ControllerParseResult = {
controller: Controller;
@@ -221,10 +222,7 @@ function nodeToElement(node: Record<string, unknown>): InputElement | undefined
const SYSTEM_UI_PACKAGES = new Set([
"android",
"com.android.systemui",
"com.google.android.inputmethod.latin",
"com.samsung.android.honeyboard",
"com.touchtype.swiftkey",
"com.microsoft.swiftkey",
...INPUT_METHOD_PACKAGES,
]);
export function isSystemUiNode(node: Record<string, unknown>): boolean {
@@ -240,6 +238,12 @@ function collectJsonElements(value: unknown, output: InputElement[], seen: Set<u
return;
}
const record = value as Record<string, unknown>;
// iOS's software keyboard is system UI too, like Android's: its keys and its Emoji and Dictate buttons are not the
// app's elements. A keyboard that rose on one side only made "Added button “Dictate”" lead a review whose real
// change was a field's label (brentvatne/euxy#172, review run 39362642343).
if (Array.isArray(record.nodes)) {
for (const node of keyboardNodes(record.nodes.filter((node): node is Record<string, unknown> => !!node && typeof node === "object" && !Array.isArray(node)))) seen.add(node);
}
const element = isSystemUiNode(record) ? undefined : nodeToElement(record);
if (element) output.push(element);
for (const [key, child] of Object.entries(record)) {
+146
View File
@@ -0,0 +1,146 @@
/**
* The system software keyboard in a controller snapshot (agent-device's flat `nodes`, which the Argent bridge also
* returns), and how long capture waits for it. Recognition is deliberately narrow: dropping one of the app's own elements
* hides a real change, which is worse than the keyboard noise this removes. A node is the keyboard's only when it, or a
* node above it, provably belongs to the system keyboard; nothing is inferred from where an element sits on the screen,
* and a `Key` alone proves nothing (React Native's `accessibilityRole="keyboardkey"` gives an app's PIN pad that type).
*/
type Rect = { x: number; y: number; width: number; height: number };
type SnapshotNode = Record<string, unknown>;
/** Android input methods: their windows name the IME's package, not the app's. */
export const INPUT_METHOD_PACKAGES: ReadonlySet<string> = new Set([
"com.google.android.inputmethod.latin",
"com.android.inputmethod.latin",
"com.samsung.android.honeyboard",
"com.touchtype.swiftkey",
"com.microsoft.swiftkey",
]);
/**
* UIKit's identifiers for the input view placeholder and the prediction bar. They mark a window as the keyboard's (see
* the Argent bridge's `keyboardWindows`); inside the app's own window an element with one of them is the app's.
*/
export const KEYBOARD_WINDOW_IDENTIFIERS: ReadonlySet<string> = new Set(["inputView", "SystemInputAssistantView"]);
/**
* The keyboard's native view classes, which agent-device's host accessibility bridge reports as `role`. The dock's Emoji
* and Dictate buttons (UIKeyboardDockItemButton) can sit outside the key plane's bounds, so they are named here.
*/
const KEYBOARD_VIEW_CLASSES = new Set(["UIKeyboardDockItemButton", "UIAccessibilityElementKBKey", "UIKeyboardLayoutStar",
"TUIPredictionView", "TUIPredictionViewCell", "_UIRemoteKeyboardPlaceholderView", "TUISystemInputAssistantView",
"TUISystemInputAssistantPageView"]);
const EDITABLE_TYPES = new Set(["textfield", "securetextfield", "searchfield", "textview"]);
const text = (value: unknown) => typeof value === "string" ? value : "";
function rectOf(node: SnapshotNode): Rect | undefined {
const rect = node.rect as Partial<Rect> | undefined;
return rect && [rect.x, rect.y, rect.width, rect.height].every(Number.isFinite) && rect.width! > 0 && rect.height! > 0 ? rect as Rect : undefined;
}
/**
* A node that belongs to the system keyboard by what it is, not where it is: XCTest's `Keyboard` element (both
* controllers), a node of a window the Argent bridge found to be the keyboard's (`systemKeyboard`), one of the keyboard's
* native view classes, or a node of an Android input method's window.
*/
function keyboardRoot(node: SnapshotNode): boolean {
return node.systemKeyboard === true || text(node.type).toLowerCase() === "keyboard" || KEYBOARD_VIEW_CLASSES.has(text(node.role))
|| INPUT_METHOD_PACKAGES.has(text(node.bundleId ?? node.packageName ?? node.package));
}
/** Every node that belongs to the system keyboard: those `keyboardRoot` names and the nodes beneath them. */
export function keyboardNodes(nodes: readonly SnapshotNode[]): Set<SnapshotNode> {
const owned = new Set<SnapshotNode>();
if (!nodes.some(keyboardRoot)) return owned;
// Only an explicit parent links a node to a keyboard above it: a node without one is no node's descendant.
const byIndex = new Map(nodes.filter(node => typeof node.index === "number").map(node => [node.index as number, node]));
for (const node of nodes) {
const seen = new Set<SnapshotNode>();
for (let current: SnapshotNode | undefined = node; current && !seen.has(current);
current = typeof current.parentIndex === "number" ? byIndex.get(current.parentIndex) : undefined) {
seen.add(current);
if (keyboardRoot(current)) { owned.add(node); break; }
}
}
return owned;
}
function union(rects: readonly Rect[]): Rect | undefined {
if (!rects.length) return undefined;
const x = Math.min(...rects.map(rect => rect.x)), y = Math.min(...rects.map(rect => rect.y));
return { x, y, width: Math.max(...rects.map(rect => rect.x + rect.width)) - x, height: Math.max(...rects.map(rect => rect.y + rect.height)) - y };
}
/**
* Where the visible keyboard is, from the nodes that span it (the `Keyboard` element, the keyboard window's input view and
* prediction bar, the native views, an input method's window), or undefined when none shows. Containers inside the
* keyboard can report a scroll extent past the screen, so not every node it owns counts.
*/
export function keyboardBounds(nodes: readonly SnapshotNode[]): Rect | undefined {
return union(nodes.filter(node => node.visible !== false && keyboardRoot(node) && (node.systemKeyboard !== true
|| text(node.type).toLowerCase() === "keyboard" || KEYBOARD_WINDOW_IDENTIFIERS.has(text(node.identifier)))).flatMap(node => rectOf(node) ?? []));
}
/**
* A text input that has the keyboard focus: agent-device reports `focused` from XCTest's keyboard focus and Android's
* focused flag; the Argent bridge passes on Argent's own.
*/
export function focusedTextInput(nodes: readonly SnapshotNode[]): boolean {
return nodes.some(node => node.focused === true && node.visible !== false
&& (EDITABLE_TYPES.has(text(node.type).toLowerCase()) || /EditText$/.test(text(node.type)) || node.editable === true));
}
/** A snapshot as the wait reads it: Argent also says whether a keyboard shows, which its Android tree cannot. */
export type KeyboardRead = { nodes: readonly SnapshotNode[]; keyboardVisible?: boolean };
/** Where the keyboard is in one read: its bounds, or only that one shows when the controller cannot place it. */
function keyboardPosition(read: KeyboardRead): string | undefined {
const bounds = keyboardBounds(read.nodes);
return bounds ? JSON.stringify(bounds) : read.keyboardVisible === true ? "visible" : undefined;
}
/**
* How capture waits for a focused field's keyboard. Reads are 250 ms apart at most, and at least two follow the first,
* so a controller whose snapshot takes a second or more still sees the keyboard rise. Waiting for a keyboard that has not
* appeared ends after a second; one that has appeared but has not yet held still is read until it does, up to six reads
* or four seconds, so a slow controller cannot keep the capture waiting.
*/
export const KEYBOARD_POLL_MS = 250;
const KEYBOARD_WAIT_MS = 1_000;
const KEYBOARD_CAP_MS = 4_000;
const KEYBOARD_MIN_READS = 2;
const KEYBOARD_MAX_READS = 6;
export type KeyboardSettle = {
/** `shown`: in the same place in two reads running. `unfocused`: the field let go and no keyboard shows. */
readonly outcome: "shown" | "moving" | "absent" | "unfocused";
readonly reads: number;
/** Whether to read the screen again, `elapsedMs` after the first read. */
more(elapsedMs: number): boolean;
read(next: KeyboardRead): void;
};
/**
* The wait for a screen whose text field has the focus, or undefined when none has it. A keyboard already up counts only
* once a later read finds it in the same place: one caught rising or going down is still moving.
*/
export function keyboardSettle(first: KeyboardRead): KeyboardSettle | undefined {
if (!focusedTextInput(first.nodes)) return undefined;
let previous = keyboardPosition(first), reads = 0;
let outcome: KeyboardSettle["outcome"] = previous ? "moving" : "absent";
return {
get outcome() { return outcome; },
get reads() { return reads; },
more: elapsedMs => (outcome === "moving" || outcome === "absent") && reads < KEYBOARD_MAX_READS
&& (reads < KEYBOARD_MIN_READS || (elapsedMs < KEYBOARD_CAP_MS && (outcome === "moving" || elapsedMs < KEYBOARD_WAIT_MS))),
read: next => {
reads += 1;
const position = keyboardPosition(next);
outcome = position && position === previous ? "shown" : position ? "moving" : focusedTextInput(next.nodes) ? "absent" : "unfocused";
previous = position;
},
};
}
+80 -16
View File
@@ -17,6 +17,7 @@ import { type CaptureSetupSnapshot, credentialSavePromptDecline, describeSetupAc
import { silentProgress } from "../shared/progress.js";
import type { CaptureSetupReady } from "../config/explore-config.js";
import { READINESS_POLL_MS, defaultReadinessSeconds } from "./readiness.js";
import { KEYBOARD_POLL_MS, keyboardSettle, type KeyboardRead, type KeyboardSettle } from "./keyboard.js";
import { CAPTURE_OBSERVER_IDENTIFIER, captureObserverMarkers, parseCaptureDestination, captureDestinationMatches } from "./observer.js";
import { type CiAgentRuntimeOptions, runCiAgentNavigation, assertAgentCaptureContext } from "./agent/runtime.js";
import { createExplorationInputs } from "../config/exploration-inputs.js";
@@ -92,6 +93,11 @@ function snapshotNodes(raw: string): Record<string, unknown>[] {
return data.nodes;
}
/** Whether the controller says a keyboard shows: Argent does, and its Android tree has no input method window to show it. */
function keyboardVisibility(data: Record<string, unknown>): { keyboardVisible?: boolean } {
return typeof data.keyboardVisible === "boolean" ? { keyboardVisible: data.keyboardVisible } : {};
}
/** The visible element each required label matched, so an image assessor can find icon-only controls. */
export function verifiedCaptureLabels(nodes: readonly Record<string, unknown>[], requiredLabels: readonly string[],
/** The screen's portrait height, when the screenshot is that screen turned a quarter clockwise (an opened iPhone Duo). */
@@ -106,6 +112,24 @@ export function verifiedCaptureLabels(nodes: readonly Record<string, unknown>[],
}));
}
/**
* The full-screen logical space a screenshot shows: the one Application node at the origin whose aspect is the image's,
* at any pixel density. Failing that, the one whose aspect is the image's transpose (`transposed`): the image is then the
* screen turned a quarter. A square screen matches both ways and is read upright. `slack` is how many image pixels the
* root, scaled to the image, may miss it by; capture asks for an exact match.
*/
export function screenSpace(nodes: readonly Record<string, unknown>[], image: { width: number; height: number }, slack = 0): { width: number; height: number; transposed: boolean } | undefined {
const roots = (transposed: boolean) => nodes.filter(node => node.type === "Application" && isRecord(node.rect)
&& node.rect.x === 0 && node.rect.y === 0 && typeof node.rect.width === "number" && typeof node.rect.height === "number"
&& Number.isFinite(node.rect.width) && Number.isFinite(node.rect.height) && node.rect.width > 0 && node.rect.height > 0
&& Math.abs(transposed ? node.rect.width * image.width - node.rect.height * image.height : node.rect.width * image.height - node.rect.height * image.width) <= slack * node.rect.height);
const upright = roots(false), turned = upright.length ? [] : roots(true);
const matched = upright.length ? upright : turned;
if (matched.length !== 1) return undefined;
const rect = matched[0]!.rect as { width: number; height: number };
return { width: rect.width, height: rect.height, transposed: turned.length === 1 };
}
/**
* A swipe that closes a presented sheet, or undefined when the snapshot does not prove a sheet.
* The route observer is drawn at the focused scene's top-left corner. A sheet's scene starts below
@@ -377,7 +401,7 @@ export async function capturePlannedRoutes(options: {
lastSnapshot = raw;
const generation = typeof data.refsGeneration === "number" ? data.refsGeneration : undefined;
const decline = declined < 2 && generation ? credentialSavePromptDecline(nodes) : undefined;
if (!decline) return { nodes, truncated: false, ...(generation ? { refsGeneration: generation } : {}) };
if (!decline) return { nodes, truncated: false, ...(generation ? { refsGeneration: generation } : {}), ...keyboardVisibility(data) };
controller(["press", `@${String(decline.ref).replace(/^@/, "")}~s${generation}`, "--level", "full", "--json"]);
recordDeclinedPrompt("deterministic");
controller(["wait", "500", "--level", "full", "--json"]);
@@ -507,6 +531,35 @@ export async function capturePlannedRoutes(options: {
.sort());
/** The controller's own quiescence gate; a timeout is reported by the snapshot comparison below. */
const settle = () => { controller(["wait", "stable", "500", "8000", "--level", "full", "--json"], true); };
/**
* A screen that focused a text field shows the keyboard that field raises, as its users see it. The keyboard rises
* after the screen has settled, so the controller's quiescence gate can pass in between: in brentvatne/euxy#172
* (review run 39362642343) the rename sheet's base was captured with its cursor showing and no keyboard, its head with
* the keyboard up, and the 48% mismatch was the keyboard. The screen is read again until the keyboard is in the same
* place twice running (see keyboardSettle); a keyboard that never comes is not waited for long, and none is dismissed.
*/
const awaitKeyboard = (route: SourceRoute, first: KeyboardRead): readonly Record<string, unknown>[] => {
const settle = keyboardSettle(first);
if (!settle) return first.nodes;
const startedAt = clock();
let current = first, pollStartedAt = startedAt;
// A read counts for at least one poll interval, so a clock that stands still (a fake controller) still expires.
const elapsed = () => Math.max(clock() - startedAt, settle.reads * KEYBOARD_POLL_MS);
while (settle.more(elapsed())) {
(capture.sleep ?? pause)(KEYBOARD_POLL_MS - (clock() - pollStartedAt));
pollStartedAt = clock();
current = snapshot();
settle.read(current);
}
recordKeyboardWait(route.id, settle, elapsed());
return current.nodes;
};
const recordKeyboardWait = (route: string | undefined, settle: KeyboardSettle, elapsedMs: number) => {
const diagnostics = join(store.root, "capture-diagnostics");
mkdirSync(diagnostics, { recursive: true, mode: 0o700 });
appendFileSync(join(diagnostics, "keyboard.jsonl"), `${JSON.stringify({ ...(route ? { route } : { phase: "agent-observe" }), outcome: settle.outcome,
reads: settle.reads, elapsedMs, at: new Date(clock()).toISOString() })}\n`, { mode: 0o600 });
};
const attempted = new Set<string>();
const captureCurrent = async (route: SourceRoute, attempt: number, before?: readonly Record<string, unknown>[],
unattributed?: NonNullable<CaptureBundle["outcomes"][string]["observed"]> & { message: string }) => {
@@ -514,9 +567,11 @@ export async function capturePlannedRoutes(options: {
attempted.add(route.id);
const temporary = await mkdtemp(join(store.root, ".capture-"));
try {
const accepted = before ?? waitForDestination(route, () => resolveDeepLinkConfirmation());
if (unattributed) assertNoAlert({ route, nodes: accepted });
else landing(accepted, route, inventory);
const arrived = before ?? waitForDestination(route, () => resolveDeepLinkConfirmation());
if (unattributed) assertNoAlert({ route, nodes: arrived });
else landing(arrived, route, inventory);
const accepted = unattributed ? arrived : awaitKeyboard(route, { nodes: arrived });
if (accepted !== arrived) landing(accepted, route, inventory);
assertSafeScreenshot(accepted);
const screenshot = join(temporary, "screen.png");
await writeFile(screenshot, "", { flag: "wx", mode: 0o600 });
@@ -663,7 +718,7 @@ export async function capturePlannedRoutes(options: {
const generation = typeof data.refsGeneration === "number" ? data.refsGeneration : undefined;
const decline = declined < 2 && generation ? credentialSavePromptDecline(nodes) : undefined;
if (!decline) {
const snapshot: CaptureSetupSnapshot = { nodes, truncated: false, ...(generation ? { refsGeneration: generation } : {}) };
const snapshot: CaptureSetupSnapshot = { nodes, truncated: false, ...(generation ? { refsGeneration: generation } : {}), ...keyboardVisibility(data) };
return { snapshot, raw };
}
await command(["press", `@${String(decline.ref).replace(/^@/, "")}~s${generation}`, "--level", "full", "--json"], signal);
@@ -693,18 +748,11 @@ export async function capturePlannedRoutes(options: {
* then the transpose of the frame, and a tap is turned back (see `toLogical`).
*/
const bounds = (frame: Candidate) => {
const roots = (transposed: boolean) => frame.snapshot.nodes.filter(node => node.type === "Application" && isRecord(node.rect)
&& node.rect.x === 0 && node.rect.y === 0 && typeof node.rect.width === "number" && typeof node.rect.height === "number"
&& Number.isFinite(node.rect.width) && Number.isFinite(node.rect.height) && node.rect.width > 0 && node.rect.height > 0
&& (transposed ? node.rect.width * frame.width === node.rect.height * frame.height : node.rect.width * frame.height === node.rect.height * frame.width));
// A square screen matches both ways; it is read upright, as before rotated frames existed.
const upright = roots(false), turned = upright.length ? [] : roots(true);
const matched = upright.length ? upright : turned;
if (matched.length !== 1) throw captureError("AGENT_COORDINATE_BOUNDS_UNVERIFIED", "The screenshot does not identify one matching full-screen logical coordinate space.");
// The official iOS screenshot is full-screen. Its default is 1x, but a proven
// uniform pixel density must still map to XCTest points rather than raw pixels.
const rect = matched[0]!.rect as { width: number; height: number };
return { width: rect.width, height: rect.height, transposed: turned.length === 1 };
const space = screenSpace(frame.snapshot.nodes, frame);
if (!space) throw captureError("AGENT_COORDINATE_BOUNDS_UNVERIFIED", "The screenshot does not identify one matching full-screen logical coordinate space.");
return space;
};
/**
* A normalized point (x, y) in the screenshot, in the tree's logical points. Upright, it scales by the root. Turned:
@@ -769,16 +817,32 @@ export async function capturePlannedRoutes(options: {
}
}
};
/** The same wait for a focused field's keyboard as a route link's capture, before an observation's screenshot. */
const awaitKeyboardAsync = async (first: CaptureSetupSnapshot, signal: AbortSignal): Promise<CaptureSetupSnapshot> => {
const settle = keyboardSettle(first);
if (!settle) return first;
const startedAt = clock();
let current = first;
const elapsed = () => Math.max(clock() - startedAt, settle.reads * KEYBOARD_POLL_MS);
while (settle.more(elapsed())) {
await sleep(KEYBOARD_POLL_MS, undefined, { signal: liveSignal(signal) });
current = (await readSnapshot(signal)).snapshot;
settle.read(current);
}
recordKeyboardWait(undefined, settle, elapsed());
return current;
};
const observe = async (signal: AbortSignal, options?: { includeImage: boolean }) => {
await discard();
await appIdentity(signal);
const first = await readSnapshotThroughDeferral(signal);
const before = first.snapshot;
let before = first.snapshot;
if (first.retries > 0) {
await appendFile(join(dirname(tracePath), "accessibility-deferral.jsonl"), `${JSON.stringify({ retries: first.retries, waitedMs: first.deferredMs,
at: new Date(clock()).toISOString() })}\n`, { mode: 0o600 });
}
if (options?.includeImage === false) return { snapshot: before };
before = await awaitKeyboardAsync(before, signal);
try { assertSafeScreenshot(before.nodes); }
catch (error) {
if (error instanceof CiError && error.code === "CAPTURE_SENSITIVE_INPUT") return { snapshot: before, imageUnavailable: { code: error.code, message: error.message } };
+3
View File
@@ -825,6 +825,9 @@ async function executeCiMap(options: CiMapOptions, setStage: (stage: string) =>
// can leave the panel it lights black, as it does for Argent: the controller then reads the old panel and a
// relaunch draws nothing. The display is the check, through the controller's own screenshot: in the pose's shape
// and drawn. If it is not, SpringBoard is restarted, the app reopened and the device folded again, once.
// The open pose's shape is taken to be landscape. That is unverified for agent-device on EAS: local agent-device
// runs returned the opened inner panel on its portrait axes (2007×2853), as explore:binary turns it from; the
// landscape shots seen on EAS (2853×2007) were Argent's, taken with its LandscapeRight rotation.
const open = (relaunch: boolean) => runner({ command: controllerCommand, args: ["open", options.bundleId, ...(relaunch ? ["--relaunch"] : []), "--session", CONTROLLER_SESSION,
...target, "--platform", options.platform, "--json"], cwd: simulatorProject, environment: controllerEnvironment, timeoutMs: 15 * 60_000 });
const shown = async () => {
+17 -5
View File
@@ -6,6 +6,8 @@ export type CaptureSetupSnapshot = {
nodes: readonly Record<string, unknown>[];
refsGeneration?: number;
truncated?: boolean;
/** Whether the controller says a software keyboard shows, when it says (Argent does). */
keyboardVisible?: boolean;
};
type Awaitable<T> = T | Promise<T>;
@@ -98,13 +100,24 @@ function failure(code: string, message: string, details?: Record<string, unknown
...(details ? { details } : {}) });
}
export const SETUP_SELECTOR_MAX = 256;
/** A label or identifier the executor can select by; a replay plan's selectors are checked against it before any device starts. */
export function isSetupSelector(value: unknown): value is string {
return typeof value === "string" && value.trim().length > 0 && value.length <= SETUP_SELECTOR_MAX && !/[\u0000-\u001f\u007f]/.test(value);
}
/** Text the executor can fill or type: one line of at most 4,096 characters. */
export function isSetupInput(value: unknown): value is string {
return typeof value === "string" && value.length <= 4_096 && !/[\u0000\r\n]/.test(value);
}
function selectorValue(selector: CaptureSetupSelector): { key: "label" | "identifier"; value: string } {
const hasLabel = typeof selector.label === "string";
const hasIdentifier = typeof selector.identifier === "string";
const key = hasLabel ? "label" : "identifier";
const value = selector[key];
if (hasLabel === hasIdentifier || typeof value !== "string" || !value.trim()
|| value.length > 256 || /[\u0000-\u001f\u007f]/.test(value)) {
if (hasLabel === hasIdentifier || !isSetupSelector(value)) {
throw failure("SETUP_SELECTOR_INVALID", "A UI selector must contain exactly one nonempty label or identifier.");
}
return { key, value };
@@ -274,8 +287,7 @@ export async function executeCaptureSetup(actions: readonly CaptureSetupAction[]
case "tap":
case "fill": {
selectorValue(action);
if (action.action === "fill" && (typeof action.text !== "string" || action.text.length > 4_096
|| /[\u0000\r\n]/.test(action.text))) {
if (action.action === "fill" && !isSetupInput(action.text)) {
throw failure("SETUP_INPUT_INVALID", "A UI fill input must be a single line of at most 4,096 characters.");
}
let resolved = false;
@@ -330,7 +342,7 @@ export async function executeCaptureSetup(actions: readonly CaptureSetupAction[]
break;
}
case "type": {
if (typeof action.text !== "string" || !action.text || action.text.length > 4_096 || /[\u0000\r\n]/.test(action.text)) {
if (!isSetupInput(action.text) || !action.text) {
throw failure("SETUP_INPUT_INVALID", "A UI type input must be a nonempty single line of at most 4,096 characters.");
}
let ready = false;
+2 -2
View File
@@ -90,10 +90,10 @@ const flagKinds: Record<string, Record<string, "value" | "boolean">> = {
"explore:binary": { archive:"value", output:"value", platform:"value", controller:"value", simulator:"value", device:"value", "bundle-id":"value", "launch-url":"value", hints:"value", "allow-destructive":"value",
"explore:binary": { replay:"value", archive:"value", output:"value", platform:"value", controller:"value", simulator:"value", device:"value", "bundle-id":"value", "launch-url":"value", hints:"value", "allow-destructive":"value",
"max-duration-minutes":"value", "max-screens":"value", "max-actions":"value", provider:"value", model:"value", owner:"value", project:"value",
upload:"boolean", "no-upload":"boolean", "expires-in":"value", alias:"value", "no-open":"boolean", port:"value" },
"explore:session": { session:"value", "env-file":"value", "bundle-id":"value", "app-name":"value", "app-version":"value", output:"value", platform:"value", controller:"value", device:"value", "launch-url":"value", hints:"value",
"explore:session": { replay:"value", session:"value", "env-file":"value", "bundle-id":"value", "app-name":"value", "app-version":"value", output:"value", platform:"value", controller:"value", device:"value", "launch-url":"value", hints:"value",
"allow-destructive":"value", "max-duration-minutes":"value", "max-screens":"value", "max-actions":"value", provider:"value", model:"value", owner:"value", project:"value",
upload:"boolean", "expires-in":"value", alias:"value", "no-open":"boolean", port:"value" },
"ci:report": { input:"value", output:"value", target:"value", platform:"value" },
+3
View File
@@ -7,6 +7,7 @@ import { assertHostingExport, deployExplorationReport, zipReportDirectory } from
import { keepReports } from "./report-commands.js";
import { warnIgnoredExpiry } from "../config/explore-config.js";
import { argumentError, flag, flags, has, nonNegativeInteger, oneOf, output, positional, positiveInteger } from "./args.js";
import { readReplayPlan } from "../binary/replay.js";
/** `--alias NAME` plus a fixed token from EXPLORE_PREVIEW_TOKEN: the hosted link stays the same from one run to the next (validated by the deploy). */
@@ -49,6 +50,7 @@ export async function exploreBinary(): Promise<void> {
...(flag("max-duration-minutes") ? { maxDurationMinutes: positiveInteger("max-duration-minutes") } : {}),
...(flag("max-screens") ? { maxScreens: positiveInteger("max-screens") } : {}),
...(flag("max-actions") ? { maxActions: positiveInteger("max-actions") } : {}),
...(flag("replay") ? { replay: await readReplayPlan(resolve(process.cwd(), flag("replay")!)) } : {}),
});
say(`exploration ${result.exploration.stopReason}: ${result.exploration.screens} screens, ${result.exploration.observations} captures, ${result.exploration.actions} actions`);
const report = await writeExplorationReport(result, join(outputDirectory, "report"));
@@ -113,6 +115,7 @@ export async function exploreSession(): Promise<void> {
...(flag("max-duration-minutes") ? { maxDurationMinutes: positiveInteger("max-duration-minutes") } : {}),
...(flag("max-screens") ? { maxScreens: positiveInteger("max-screens") } : {}),
...(flag("max-actions") ? { maxActions: positiveInteger("max-actions") } : {}),
...(flag("replay") ? { replay: await readReplayPlan(resolve(process.cwd(), flag("replay")!)) } : {}),
});
say(`exploration ${result.exploration.stopReason}: ${result.exploration.screens} screens, ${result.exploration.observations} captures, ${result.exploration.actions} actions`);
const report = await writeExplorationReport(result, join(outputDirectory, "report"));
+1
View File
@@ -56,6 +56,7 @@ const BINARY = `Prebuilt apps (no source needed; runs with your local claude or
and the model and the report only see the reference)
--max-duration-minutes N --max-screens N --max-actions N --provider anthropic|openai --model ID
--owner ACCOUNT --project SLUG --allow-destructive "Label,Other label"
--replay FILE (takes the steps in FILE instead of a model's, so every run and device takes the same path; no model access needed, and not with --provider, --model or --hints)
--alias NAME (one EAS Hosting host per app that always serves the newest run; with EXPLORE_PREVIEW_TOKEN
set to a fixed 32-character token, the hosted link is the same every run)
(iOS needs a simulator build; a device .ipa and Android .aab are rejected before any device starts)
+6 -1
View File
@@ -178,7 +178,7 @@ export type SystemRegion = { kind: "status-bar" | "home-indicator"; x: number; y
* EAS Simulator captures of both panels; the pill is the same 40×64 points on each, only its top differs. The app's own
* rail controls start about 3% of the height below it on the outer panel, so the padding stays small.
*/
const DUO_PANELS = {
export const DUO_PANELS = {
outer: { width: 466, height: 678, statusBar: { x: 398, y: 87, width: 40, height: 64 } },
inner: { width: 951, height: 669, statusBar: { x: 883, y: 34, width: 40, height: 64 } },
} as const;
@@ -251,6 +251,11 @@ export type PublicSurfaceReview = {
description: string;
/** Every deterministic accessibility fact for this surface, in reviewer priority order. */
facts?: Array<{ category: string; templateId: string; bullet: string }>;
/**
* The screenshots match because the change is past the edge of the screen: every element its facts name reaches
* beyond the captured screen. Only on a `modified` verdict whose screenshots match.
*/
offScreen?: true;
};
/** An evidence-bound explanation. It never changes a deterministic comparison verdict. */
export type PublicChangeNote = {
+1
View File
@@ -57,6 +57,7 @@ function assertReviewFields(surface: Surface, enforceComparisonEvidence: boolean
|| (["modified", "unchanged", "variant"].includes(review.status) && (!surface.base || !surface.head))
|| (surface.sourceScope !== undefined && !["added", "removed"].includes(review.status)
&& !SCOPE_VERDICTS[surface.sourceScope]?.includes(review.status))
|| (review.offScreen !== undefined && (review.offScreen !== true || review.status !== "modified" || surface.visual?.verdict !== "unchanged"))
|| (review.facts !== undefined && (!Array.isArray(review.facts) || review.facts.length > 100
|| review.facts.some((fact) => !fact || typeof fact.category !== "string" || typeof fact.templateId !== "string"
|| typeof fact.bullet !== "string" || !fact.bullet.trim() || fact.bullet.length > MAX_REVIEW_DESCRIPTION)))) {
+6 -2
View File
@@ -6,6 +6,7 @@ import {
type Observation,
type Screen,
} from "../types.js";
import { accessibilityFingerprintsComparable } from "./store.js";
/**
* Cross-run screen matching for PR-style comparisons (baseline = base branch map,
@@ -253,7 +254,8 @@ function matchEntry(match: Match): { entry: BuildComparisonEntry; review: Compar
// captured on both sides) hide a real change to the primary content.
const baselineHash = match.baseline.representative.fingerprints.accessibility;
const currentHash = match.current.representative.fingerprints.accessibility;
const verdict = baselineHash && currentHash
// Hashes from different fingerprint versions (an older Explore's map) say nothing about whether the trees differ.
const verdict = baselineHash && currentHash && accessibilityFingerprintsComparable(baselineHash, currentHash)
? (baselineHash === currentHash ? "unchanged" : "modified")
: "unknown";
const unchanged = verdict === "unchanged";
@@ -262,7 +264,9 @@ function matchEntry(match: Match): { entry: BuildComparisonEntry; review: Compar
? `Matched by ${signals}; the representative captures share an accessibility fingerprint, so the rendered tree is unchanged.`
: verdict === "modified"
? `Matched by ${signals}; the representative captures have different accessibility fingerprints, so the rendered tree changed.`
: `Matched by ${signals}; accessibility fingerprints are unavailable on at least one side, so the change state could not be verified from fingerprints.`;
: baselineHash && currentHash
? `Matched by ${signals}; the representative captures were fingerprinted by different Explore versions, so the change state could not be verified from fingerprints.`
: `Matched by ${signals}; accessibility fingerprints are unavailable on at least one side, so the change state could not be verified from fingerprints.`;
const entry: BuildComparisonEntry = {
id: `match-${match.current.screen.id}`,
relationship: "one-to-one",
+23 -5
View File
@@ -14,6 +14,7 @@ import { dirname, extname, isAbsolute, join, relative, resolve, sep } from "node
import { fileURLToPath } from "node:url";
import { Ajv2020, type ErrorObject, type ValidateFunction } from "ajv/dist/2020.js";
import { isSystemUiNode } from "../capture/controller-output.js";
import { keyboardNodes } from "../capture/keyboard.js";
import { validateNavigationGraph } from "../contracts/navigation-graph.js";
import {
CONTRACT,
@@ -67,11 +68,26 @@ export const normalizeSelector = (input?: string): string | undefined => {
return input.replace(/\s+/g, " ").trim() || undefined;
};
const VOLATILE_SNAPSHOT_NODE_KEYS = new Set(["ref"]);
/** A ref names a node within one read; which field has the focus can change between two reads of one screen. */
const VOLATILE_SNAPSHOT_NODE_KEYS = new Set(["ref", "focused"]);
/**
* The accessibility fingerprint's algorithm, written as its prefix. Version 2 leaves the system keyboard and the focus out
* of the tree; a hash from before carries no prefix. Two fingerprints of different versions say nothing about whether the
* trees differ, so a comparison treats them as not comparable rather than changed (see accessibilityFingerprintsComparable).
*/
const ACCESSIBILITY_FINGERPRINT_VERSION = "a2";
export function accessibilityFingerprintsComparable(left: string, right: string): boolean {
const version = (hash: string) => /^(a\d+):/.exec(hash)?.[1] ?? "a1";
return version(left) === version(right);
}
/** Canonical JSON of the node tree only: controller wrappers carry per-session diagnostics and
* ref generations that would otherwise make identical screens fingerprint differently, and Android
* system UI nodes (status bar clock and signal, keyboard) change between emulator sessions. */
* system UI nodes (status bar clock and signal, keyboard) change between emulator sessions. The
* system keyboard on either platform is left out as the elements leave it out, so a keyboard up on
* one side only does not make two captures of one screen differ. */
function normalizeJsonSnapshotNodes(input: string): string | undefined {
if (!/^\s*\{/.test(input)) return undefined;
let value: unknown;
@@ -86,10 +102,12 @@ function normalizeJsonSnapshotNodes(input: string): string | undefined {
if (!data || typeof data !== "object" || !Array.isArray((data as { nodes?: unknown }).nodes)) {
return undefined;
}
const nodes = (data as { nodes: unknown[] }).nodes;
const keyboard: Set<unknown> = keyboardNodes(nodes.filter((entry): entry is Record<string, unknown> => !!entry && typeof entry === "object" && !Array.isArray(entry)));
const canonical = (node: unknown): unknown => {
if (Array.isArray(node)) {
return node
.filter((entry) => !(entry && typeof entry === "object" && isSystemUiNode(entry as Record<string, unknown>)))
.filter((entry) => !(entry && typeof entry === "object" && (isSystemUiNode(entry as Record<string, unknown>) || keyboard.has(entry))))
.map(canonical);
}
if (!node || typeof node !== "object") return node;
@@ -100,7 +118,7 @@ function normalizeJsonSnapshotNodes(input: string): string | undefined {
.map(([key, entry]) => [key, canonical(entry)]),
);
};
return `nodes\0${JSON.stringify(canonical((data as { nodes: unknown[] }).nodes))}`;
return `nodes\0${JSON.stringify(canonical(nodes))}`;
}
export const normalizeSnapshot = (input: string): string =>
@@ -118,7 +136,7 @@ export async function fingerprintState(options: {
snapshotText?: string;
}): Promise<ObservationFingerprints> {
const accessibility = options.snapshotText
? sha256(`snapshot\0${normalizeSnapshot(options.snapshotText)}`)
? `${ACCESSIBILITY_FINGERPRINT_VERSION}:${sha256(`snapshot\0${normalizeSnapshot(options.snapshotText)}`)}`
: undefined;
const screenshot = options.screenshotPath ? await readFile(options.screenshotPath) : undefined;
const visual = screenshot ? sha256(Buffer.concat([Buffer.from("visual\0"), screenshot])) : undefined;
+23 -8
View File
@@ -254,24 +254,39 @@ function explorationScreenItem(context: AgentContext, map: Json, screen: Json):
viewerUrl: ID.test(id) ? viewerUrl(context, { screen: id }) : undefined,
});
}
function screenshotUrl(context: AgentContext, map: Json, screen: Json): string | undefined {
/** The screenshot a screen is shown by: its latest observation's. */
function screenshotArtifact(map: Json, screen: Json): Json | undefined {
const observationId = strings(screen.observationIds).at(-1);
const observation = arr(map.observations).map(rec).find(o => o.id === observationId);
const artifact = arr(map.artifacts).map(rec).find(a => a.id === observation?.screenshotArtifactId);
const sha = str(artifact?.sha256);
return arr(map.artifacts).map(rec).find(a => a.id === observation?.screenshotArtifactId);
}
function screenshotUrl(context: AgentContext, map: Json, screen: Json): string | undefined {
const sha = str(screenshotArtifact(map, screen)?.sha256);
return sha && SHA256.test(sha) ? withPlatform(context, apiUrl(context, `/assets/${sha}`)) : undefined;
}
const elementItem = (element: Json): Json => defined({
id: str(element.id ?? element.elementId), label: str(element.label), role: str(element.role), value: str(element.value), identifier: str(element.identifier),
bounds: element.bounds === undefined ? undefined : rec(element.bounds), actions: strings(element.actions), risk: str(element.risk), actionable: bool(element.actionable), enabled: bool(element.enabled), visible: bool(element.visible),
});
/**
* One element, its bounds on the axes of the screenshot it is listed beside. A screenshot stored turned a quarter
* clockwise (an opened iPhone Duo's) is `turnedHeight` points tall on the tree's axes, where the map keeps the bounds:
* there the portrait point (x, y) shows at (H - y, x).
*/
const elementItem = (element: Json, turnedHeight?: number): Json => {
const bounds = element.bounds === undefined ? undefined : rec(element.bounds);
const [x, y, width, height] = [num(bounds?.x), num(bounds?.y), num(bounds?.width), num(bounds?.height)];
const shown = bounds && turnedHeight !== undefined && bounds.unit === "points" && x !== undefined && y !== undefined && width !== undefined && height !== undefined
? { x: turnedHeight - y - height, y: x, width: height, height: width, unit: "points" } : bounds;
return defined({
id: str(element.id ?? element.elementId), label: str(element.label), role: str(element.role), value: str(element.value), identifier: str(element.identifier),
bounds: shown, actions: strings(element.actions), risk: str(element.risk), actionable: bool(element.actionable), enabled: bool(element.enabled), visible: bool(element.visible),
});
};
const transitionItem = (transition: Json): Json => defined({
id: str(transition.id), from: str(transition.fromScreenId), to: str(transition.toScreenId), actionKind: str(transition.actionKind), elementId: str(transition.elementId), attempts: strings(transition.attemptIds).length,
});
function explorationScreenDetail(context: AgentContext, map: Json, screen: Json): Json {
const id = str(screen.id);
const transitions = arr(map.transitions).map(rec);
const elements = arr(map.elements).map(rec).filter(element => element.screenId === id).map(elementItem);
const turnedHeight = num(rec(screenshotArtifact(map, screen)?.turned).height);
const elements = arr(map.elements).map(rec).filter(element => element.screenId === id).map(element => elementItem(element, turnedHeight));
const variants = arr(map.variants).map(rec).filter(variant => variant.screenId === id).map(variant => defined({ id: str(variant.id), key: str(variant.key), name: str(variant.name), observations: strings(variant.observationIds).length }));
const gaps = arr(map.gaps).map(rec).filter(gap => gap.screenId === id).map(gapItem);
return defined({
+29 -4
View File
@@ -143,6 +143,16 @@ function notComparedChange(surface: Surface): boolean {
return verified && (!surface.review || surface.visual?.reason === "diff-failed");
}
/**
* An accessibility-only change whose elements lie past the edge of the screen (the review's `offScreen`), so another
* target's row can say the change is there, off-screen, rather than missing. Like notComparedChange, a comparison that
* failed or a redirect is not one.
*/
function offScreenChange(surface: Surface): boolean {
return surface.review?.offScreen === true && isAccessibilityOnlyChange(surface) && surface.visual?.reason !== "diff-failed"
&& (surface.head ?? surface.base)?.kind !== "redirect";
}
function routeOf(surface: Surface): string {
return plainText(surface.head?.route || surface.base?.route || surface.id, MAX_ROUTE_LENGTH);
}
@@ -302,7 +312,8 @@ export function platformRecordOf(state: PublicCommentState, target?: CommentTarg
const group = groupOf(surface, evidenceBased);
if (group === "unlisted") continue;
const scheme = schemeOf(surface.id);
record.changes.push({ id: surface.id, route: routeOf(surface), group, description: descriptionOf(surface, group), ...(scheme ? { scheme } : {}) });
record.changes.push({ id: surface.id, route: routeOf(surface), group, description: descriptionOf(surface, group), ...(scheme ? { scheme } : {}),
...(group === "accessibility-only" && offScreenChange(surface) ? { offScreen: true as const } : {}) });
}
const navigation = state.map ? navigationOf(state.review, new Map(record.changes.map(change => [change.id, change.group]))) : undefined;
if (navigation) record.navigation = navigation;
@@ -562,10 +573,24 @@ function renderChangeRow(row: ChangeRow, ready: PlatformRecord[]): string {
text = `${route} · ${description}`;
}
const tag = row.group === "accessibility-only" ? "accessibility only" : row.group === "needs-review" ? "needs review" : undefined;
return `- ${[text, tag, ...whereNote(row, ready)].filter(Boolean).join(" · ")}`;
}
/**
* Where a screen's change shows. A target whose screenshots match because the changed elements lie past the edge of the
* screen has the change, off-screen, so it is named apart: "iPhone 17, Android only · off-screen on iPhone Duo (closed)"
* for Tempo's footnote below the fold of its sheet in brentvatne/euxy#172 (review run 39362642343). The targets that show
* it are still named the shorter way among the others (none when all of them do); without such a target the note is what
* it always was.
*/
function whereNote(row: ChangeRow, ready: PlatformRecord[]): string[] {
const offScreen = row.group === "modified"
? row.targets.filter(record => record.changes?.some(change => change.id === row.id && change.offScreen === true)) : [];
const shown = row.targets.filter(record => !offScreen.includes(record));
// A target with screens it has no verdict on may have changed this one too: name where it changed, not where it did not.
const where = ready.some(record => !row.targets.includes(record) && gaps(record).length)
? `on ${row.targets.map(labelOf).join(", ")}` : targetsNote(row.targets, ready);
return `- ${[text, tag, where].filter(Boolean).join(" · ")}`;
const unsure = ready.some(record => !row.targets.includes(record) && gaps(record).length);
const visible = unsure ? `on ${shown.map(labelOf).join(", ")}` : targetsNote(shown, ready.filter(record => !offScreen.includes(record)));
return [visible, offScreen.length ? `off-screen on ${offScreen.map(labelOf).join(", ")}` : undefined].filter((part): part is string => Boolean(part));
}
/** "Android only" / "not on Android": the targets a screen changed on, when not all of them, named the shorter way. */
+7 -3
View File
@@ -103,8 +103,11 @@ export type PlatformRecord = {
accessibility?: { compared: number; described: number; failed: number };
routes?: number;
};
/** The review's changed surfaces, one per surface; `scheme` for a secondary appearance (`<route>@<scheme>`). */
changes?: { id: string; route: string; group: ChangeGroup; description: string; scheme?: string }[];
/**
* The review's changed surfaces, one per surface; `scheme` for a secondary appearance (`<route>@<scheme>`). `offScreen`
* marks an accessibility-only change whose elements lie past the edge of the screen.
*/
changes?: { id: string; route: string; group: ChangeGroup; description: string; scheme?: string; offScreen?: true }[];
/** Screens whose source changed and whose captures were verified, but that got no verdict: the comparison did not run or failed. */
notCompared?: number;
/** Differences stated once rather than per screen (variants, app-wide source changes). */
@@ -270,7 +273,8 @@ export function parsePlatformRecord(value: unknown): PlatformRecord | undefined
if (Array.isArray(value.changes)) {
const changes = value.changes.flatMap((entry): NonNullable<PlatformRecord["changes"]> => isRecord(entry) && isString(entry.id, 220) && isString(entry.route, 2000)
&& CHANGE_GROUPS.includes(entry.group as ChangeGroup) && isString(entry.description, 2000)
? [{ id: entry.id, route: entry.route, group: entry.group as ChangeGroup, description: entry.description, ...(isString(entry.scheme, 20) ? { scheme: entry.scheme } : {}) }] : []);
? [{ id: entry.id, route: entry.route, group: entry.group as ChangeGroup, description: entry.description, ...(isString(entry.scheme, 20) ? { scheme: entry.scheme } : {}),
...(entry.offScreen === true && entry.group === "accessibility-only" ? { offScreen: true as const } : {}) }] : []);
if (changes.length <= 2000) record.changes = changes;
}
const navigation = parseNavigation(value.navigation);
+1
View File
@@ -716,6 +716,7 @@ export async function assessPublicEvidence(options: {
const summary = summarizeReviewerChanges({ changes: result.reviewerChanges });
accessibilityCounts.compared += 1;
review = decideReview(scope, visual, summary);
if (result.offScreen && review.status === "modified" && visual.verdict === "unchanged") review = { ...review, offScreen: true };
if (review.facts?.length) accessibilityCounts.described += 1;
} catch (error) {
accessibilityCounts.failed += 1;
+52
View File
@@ -93,6 +93,8 @@ export type EvidenceComparatorResult = {
visualAssessment?: VisualEvidenceAssessment;
/** Feed directly to summarizeReviewerChanges. */
reviewerChanges: ReviewerChangeInput[];
/** The screenshots match, and every element the changes name lies at least partly past the edge of the screen. */
offScreen?: true;
};
type SupportedInput = Extract<ReviewerChangeInput, { assessment: "supported" }>;
@@ -617,6 +619,54 @@ function outermost(elements: readonly ObservedElement[]): ObservedElement[] {
});
}
/**
* The screen in its accessibility tree's units: the element at the origin with the screenshot's shape (either way round,
* since a frame turned to show an opened iPhone Duo upright keeps the tree on the panel's portrait axes), the largest if
* several. Both controllers report the app's root that way. Without the screenshot's size there is no telling which
* element is the screen.
*/
function viewportOf(side: ResolvedSide): ElementBounds | undefined {
const image = side.map.artifacts.find(artifact => artifact.id === side.observation.screenshotArtifactId);
if (!image?.width || !image.height) return undefined;
const { width, height } = image;
const shaped = (bounds: ElementBounds) => Math.abs(bounds.width * height - bounds.height * width) <= 0.01 * width * height
|| Math.abs(bounds.width * width - bounds.height * height) <= 0.01 * width * height;
return side.observation.elements.flatMap(element => boundsOf(element) ?? [])
.filter(bounds => Math.abs(bounds.x) < 1 && Math.abs(bounds.y) < 1 && bounds.width > 0 && bounds.height > 0 && shaped(bounds))
.sort((a, b) => b.width * b.height - a.width * a.height)[0];
}
/**
* Whether the screenshots cannot show the changes because the elements they name lie past the edge of the screen: each
* change names at least one element, and more than half of each named element's area is beyond the screen on its side.
* Both controllers leave an element wholly outside the screen out of the tree (the Argent bridge as agent-device does),
* so a change past the edge can only show as an element straddling it, as Tempo's footnote below the fold of its sheet
* on the closed iPhone Duo in brentvatne/euxy#172 (review run 39362642343) would. Mostly outside, not merely crossing
* an edge: a control nudged a point or two past the edge is still on the screen, and its change stays "accessibility
* only". A change whose elements have no bounds never qualifies.
*/
function changesOffScreen(facts: readonly SupportedInput[], base: ResolvedSide, head: ResolvedSide): boolean {
const changes = facts.filter(fact => !["unstable-text", "no-visible-change", "visual-changed"].includes(fact.change.kind));
if (!changes.length) return false;
const viewports = { base: viewportOf(base), head: viewportOf(head) };
const outside = (reference: string) => {
const [, side, id] = /^(base|head):element:(.+)$/.exec(reference) ?? [];
if (side !== "base" && side !== "head") return undefined;
const viewport = viewports[side];
const element = (side === "base" ? base : head).observation.elements.find(candidate => candidate.elementId === id);
const bounds = element && boundsOf(element);
if (!viewport || !bounds) return false;
const visibleWidth = Math.max(0, Math.min(bounds.x + bounds.width, viewport.x + viewport.width) - Math.max(bounds.x, viewport.x));
const visibleHeight = Math.max(0, Math.min(bounds.y + bounds.height, viewport.y + viewport.height) - Math.max(bounds.y, viewport.y));
const area = bounds.width * bounds.height;
return area > 0 && visibleWidth * visibleHeight < area / 2;
};
return changes.every(fact => {
const named = fact.evidenceRefs.map(outside).filter((value): value is boolean => value !== undefined);
return named.length > 0 && named.every(Boolean);
});
}
function specificByLabel(elements: readonly ObservedElement[]): ObservedElement[] {
const byLabel = new Map<string, ObservedElement>();
const unlabelled: ObservedElement[] = [];
@@ -1276,6 +1326,7 @@ function commonResult(input: CommonSurfaceComparatorInput): EvidenceComparatorRe
const status: "modified" | "unchanged" = hasRuntimeDelta || reasons.length > 0
? "modified"
: "unchanged";
const offScreen = input.visualAssessment.status === "unchanged" && changesOffScreen(facts, base, head);
return {
surfaceId: input.impact.id,
status,
@@ -1288,6 +1339,7 @@ function commonResult(input: CommonSurfaceComparatorInput): EvidenceComparatorRe
),
visualAssessment: input.visualAssessment,
reviewerChanges,
...(offScreen ? { offScreen: true as const } : {}),
};
}
+36
View File
@@ -144,3 +144,39 @@ export function assertDecodedPng(contents: Buffer, label: string): void {
if (decoded[row * (rowBytes + 1)]! > 4) throw new Error(`${label} has an invalid PNG row filter`);
}
}
/** The chunks that say how a PNG's samples map to colour; pngjs reads past them and writes none of its own. */
const COLOUR_CHUNKS = new Set(["iCCP", "sRGB", "gAMA", "cHRM"]);
/**
* The image turned a quarter clockwise. An opened iPhone Duo shows a landscape UI on a panel whose native axes are
* portrait, and its screenshot comes back on those axes; turned, it is what a person sees. The source's colour chunks
* are carried over unchanged. pngjs reads every PNG as 8-bit RGBA, so a 16-bit image comes out 8-bit; simulator
* screenshots are 8-bit already.
*/
export function turnedClockwise(png: Buffer): Buffer {
const { PNG } = createRequire(import.meta.url.replace(/\?.*$/, ""))("pngjs") as { PNG: { new(size: { width: number; height: number }): DecodedFrame; sync: {
read(bytes: Buffer, options: { checkCRC: boolean }): DecodedFrame; write(frame: DecodedFrame): Buffer } } };
const source = PNG.sync.read(png, { checkCRC: false });
const { width, height } = source;
const turned = new PNG({ width: height, height: width });
for (let y = 0; y < height; y += 1) {
for (let x = 0; x < width; x += 1) {
const from = (y * width + x) * 4;
source.data.copy(turned.data, (x * height + (height - 1 - y)) * 4, from, from + 4);
}
}
const written = PNG.sync.write(turned);
// Each chunk is copied whole, CRC included: its data does not depend on the pixels or their size. They belong before
// the image data, so they go right after the header (8 bytes of signature, then IHDR's 25). The output is RGBA, so a
// grey source's ICC profile, which describes grey samples only, is left out.
const grey = png[25] === 0 || png[25] === 4;
const colour: Buffer[] = [];
for (let offset = 8; offset + 12 <= png.length;) {
const length = png.readUInt32BE(offset), type = png.toString("latin1", offset + 4, offset + 8), next = offset + 12 + length;
if (type === "IDAT" || type === "IEND" || next > png.length) break;
if (COLOUR_CHUNKS.has(type) && !(grey && type === "iCCP")) colour.push(png.subarray(offset, next));
offset = next;
}
return colour.length ? Buffer.concat([written.subarray(0, 33), ...colour, written.subarray(33)]) : written;
}
+6
View File
@@ -121,6 +121,12 @@ export type Artifact = {
mimeType: string;
width?: number;
height?: number;
/**
* A screenshot stored turned a quarter clockwise from its accessibility tree's axes (an opened iPhone Duo's landscape UI
* on its inner panel's portrait axes): the tree's screen, `width` × `height` points. Element bounds stay on the tree's
* axes, where the portrait point (x, y) shows at (height - y, x).
*/
turned?: { width: number; height: number };
createdAt: string;
};
+2
View File
@@ -201,6 +201,8 @@ test("Android fill uses Argent's fill and requires it to verify the text", async
const snapshot = await run(["snapshot", ...flags("android")]);
const node = snapshot.json.data.nodes.find(item => item.type === "TextField");
assert.ok(node, "the rename fixture has an EditText");
assert.equal(node.focused, true, "the focused field is marked, as agent-device marks it");
assert.equal(snapshot.json.data.keyboardVisible, false, "whether a keyboard shows, which the Android tree cannot say, is passed on");
assert.equal((await run(["fill", node.ref, ...flags("android"), "--", "Groove"])).exitCode, 0);
assert.deepEqual(calls.find(call => call.tool === "fill").args, { udid: "emulator-5554", text: "Groove" });
const unverified = await run(["fill", node.ref, ...flags("android"), "--", "Groove"]);
+327 -1
View File
@@ -8,7 +8,8 @@ import { tsImport } from "tsx/esm/api";
const { runBlackBoxExploration, screenEvidenceKey, elementsOf, fieldName, systemDialogTitle } = await tsImport("../../src/binary/explorer.ts", import.meta.url);
const { ExploreStore, validateMap } = await tsImport("../../src/map/store.ts", import.meta.url);
const { CONTRACT } = await tsImport("../../src/types.ts", import.meta.url);
const { assertAppUrl, describeForeground, foreignNode, guardPositionals, leftForeground, retryWhileBusy } = await tsImport("../../src/binary/run.ts", import.meta.url);
const tsRun = await tsImport("../../src/binary/run.ts", import.meta.url);
const { assertAppUrl, describeForeground, foreignNode, guardPositionals, leftForeground, retryWhileBusy } = tsRun;
const PNG = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==", "base64");
const SCREENS = {
@@ -881,3 +882,328 @@ test("when nothing passes preflight, the first failure is reported", async () =>
const createSession = (config, { auth }) => ({ async preflight() { throw new Error(`${config.provider} ${auth} refused`); } });
await assert.rejects(preflightModelSession({ createSession }, {}, () => {}), /anthropic local-login refused/);
});
test("a landscape frame on a portrait panel (an opened iPhone Duo) is stored upright", async () => {
const { createRequire } = await import("node:module");
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
const raw = new Png({ width: 1, height: 2 });
raw.data.set([255, 0, 0, 255, 0, 0, 255, 255]);
const png = Png.sync.write(raw);
const { map, root } = await harness([["observe", { includeImage: true }], ["capture", { frameId: "frame-0", title: "Home" }], ["finish", { summary: "One." }]], {
observe: async includeImage => ({ nodes: SCREENS.home, capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png, width: 1, height: 2, landscape: true } : {}) }) });
const artifact = map.artifacts.find(item => item.kind === "screenshot");
assert.deepEqual([artifact.width, artifact.height], [2, 1]);
const { readFile } = await import("node:fs/promises");
const stored = Png.sync.read(await readFile(join(root, artifact.path)));
// The panel's top (red) is the UI's right-hand side once turned a quarter clockwise.
assert.deepEqual([...stored.data], [0, 0, 255, 255, 255, 0, 0, 255]);
});
/** A PNG of `width` × `height` RGBA pixels, each given as [r, g, b, a] row by row. */
async function pngOf(width, height, pixels) {
const { createRequire } = await import("node:module");
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
const image = new Png({ width, height });
image.data.set(pixels.flat());
return Png.sync.write(image);
}
test("an opened Duo's frame shows the same pixels as a screen already seen once turned, so it is that screen (AudioScape's playlist title, rotated)", async () => {
const tabs = [{ type: "Button", label: "Home", hittable: true }, { type: "Button", label: "Library", hittable: true }];
const library = [{ type: "StaticText", label: "Library" }, { type: "Button", label: "My Playlist", hittable: true }, ...tabs];
const trees = { library, titled: [{ type: "StaticText", label: "My Playlist" }, ...tabs], untitled: [...tabs] };
const images = { library: await pngOf(1, 2, [[255, 0, 0, 255], [0, 0, 255, 255]]), playlist: await pngOf(1, 2, [[0, 255, 0, 255], [0, 0, 255, 255]]) };
let visits = 0; let frames = 0; let where = "library";
const { map } = await harness([
["observe", { includeImage: true }], ["capture", { frameId: "frame-0", title: "Library" }],
["act", { action: "tap", label: "My Playlist" }], ["observe", { includeImage: true }],
["act", { action: "back" }], ["observe", { includeImage: true }],
["act", { action: "tap", label: "My Playlist" }], ["observe", { includeImage: true }],
["finish", { summary: "Done." }],
], { observe: async includeImage => ({ nodes: trees[where], capturedAt: new Date().toISOString(),
...(includeImage ? { frameId: `frame-${frames++}`, png: Buffer.from(images[where === "library" ? "library" : "playlist"]), width: 1, height: 2, landscape: true } : {}) }),
act: async action => { if (action.action === "back") where = "library"; else where = visits++ === 0 ? "titled" : "untitled"; } });
assert.equal(map.screens.length, 2, "the second read of the playlist did not mint a screen");
assert.equal(map.observations.filter(item => item.screenId === map.screens[1].id).length, 2, "both reads are observations of the one playlist screen");
assert.equal(map.artifacts.filter(item => item.kind === "screenshot").length, 2);
});
test("a coordinate tap on an opened Duo is named by where it shows in the upright screenshot", async () => {
// The inner panel's portrait axes: a button near the portrait top right is at the upright bottom right.
const home = [
{ type: "Application", label: "", rect: { x: 0, y: 0, width: 669, height: 951 } },
{ type: "Button", label: "", hittable: true, rect: { x: 600, y: 40, width: 44, height: 44 } },
];
let frames = 0; let tapped = false;
// The look without an image carries no landscape flag; the screen's stored screenshot still says it was turned.
const { map } = await harness([
["observe", { includeImage: true }], ["capture", { frameId: "frame-0", title: "Home" }], ["observe", { includeImage: false }],
["act", { action: "tap", frameId: "frame-0", x: 622 / 669, y: 62 / 951 }],
["finish", { summary: "Done." }],
], { observe: async includeImage => ({ nodes: tapped ? SCREENS.settings : home, capturedAt: new Date().toISOString(),
...(includeImage ? { frameId: `frame-${frames++}`, png: await pngOf(1, 2, [[255, 0, 0, 255], [0, 0, 255, 255]]), width: 1, height: 2, landscape: true } : {}) }),
act: async () => { tapped = true; } });
assert.equal(map.actionAttempts.find(item => item.action.kind === "press").action.label, "icon button, bottom right");
assert.deepEqual(map.artifacts.find(item => item.kind === "screenshot").turned, { width: 669, height: 951 });
});
test("turnedClockwise puts the portrait top on the right and keeps the colour chunks", async () => {
const { turnedClockwise, assertDecodedPng } = await tsImport("../../src/shared/png-evidence.ts", import.meta.url);
const { createRequire } = await import("node:module");
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
const { crc32 } = await import("node:zlib");
// 3 wide and 2 tall; each pixel's red is its index, so where it lands says where it came from.
const plain = await pngOf(3, 2, [0, 1, 2, 3, 4, 5].map(index => [index * 40, 0, 0, 255]));
const chunk = (type, data) => {
const body = Buffer.concat([Buffer.from(type, "latin1"), data]), out = Buffer.alloc(body.length + 8);
out.writeUInt32BE(data.length, 0); body.copy(out, 4); out.writeUInt32BE(crc32(body), body.length + 4);
return out;
};
const gamma = Buffer.alloc(4); gamma.writeUInt32BE(45455);
const source = Buffer.concat([plain.subarray(0, 33), chunk("sRGB", Buffer.from([0])), chunk("gAMA", gamma), plain.subarray(33)]);
const turned = turnedClockwise(source);
assertDecodedPng(turned, "Turned");
const decoded = Png.sync.read(turned);
assert.deepEqual([decoded.width, decoded.height], [2, 3]);
// Source (x, y) lands at (height - 1 - y, x): the top row (0, 1, 2) runs down the right-hand column.
const red = [...Array(6).keys()].map(index => decoded.data[index * 4] / 40);
assert.deepEqual(red, [3, 0, 4, 1, 5, 2]);
const types = [];
for (let offset = 8; offset < turned.length; offset += 12 + turned.readUInt32BE(offset)) types.push(turned.toString("latin1", offset + 4, offset + 8));
assert.deepEqual(types.slice(0, 3), ["IHDR", "sRGB", "gAMA"]);
assert.ok(turned.includes(gamma), "the gamma is the source's");
});
test("only an opened Duo's frame on its inner panel's portrait axes is turned upright", () => {
const { duoScreenshotTurn } = tsRun;
const root = (width, height) => ({ type: "Application", label: "Expo Go", rect: { x: 0, y: 0, width, height } });
const frame = (deviceModel, nodes, width, height, platform = "ios") => duoScreenshotTurn({ platform, deviceModel, nodes, width, height });
const cases = [
["Duo open", frame("iPhone Duo", [root(669, 951)], 2007, 2853), "turn"],
["Duo half-open", frame("iPhone Duo", [root(669, 951), { type: "Button", label: "Home", rect: { x: 20, y: 40, width: 44, height: 44 } }], 2007, 2853), "turn"],
["Duo closed", frame("iPhone Duo", [root(466, 678)], 1398, 2034), "keep"],
["iPhone 17 Pro", frame("iPhone 17 Pro", [root(402, 874)], 1206, 2622), "keep"],
["an iPhone in landscape", frame("iPhone 17 Pro", [root(874, 402)], 2622, 1206), "keep"],
["iPad", frame("iPad Pro 13-inch (M5)", [root(1032, 1376)], 2064, 2752), "keep"],
["Android", frame("Pixel 9", [root(669, 951)], 2007, 2853, "android"), "keep"],
["an empty tree", frame("iPhone Duo", [], 2007, 2853), "keep"],
["a first node that isn't the Application", frame("iPhone Duo", [{ type: "Window", rect: { x: 0, y: 0, width: 300, height: 200 } }, root(669, 951)], 2007, 2853), "turn"],
["a root off by a fraction of a point", frame("iPhone Duo", [root(669.33, 951)], 2008, 2853), "turn"],
["an already-landscape image", frame("iPhone Duo", [root(669, 951)], 2853, 2007), "keep"],
["the inner panel on its own axes", frame("iPhone Duo", [root(951, 669)], 2853, 2007), "keep"],
["a Duo root on neither panel", frame("iPhone Duo", [root(700, 1000)], 2100, 3000), "unmatched"],
["a Duo root the image does not show", frame("iPhone Duo", [root(669, 951)], 2000, 2000), "unmatched"],
];
for (const [name, actual, expected] of cases) assert.equal(actual, expected, name);
});
test("the agent API lists a turned screenshot's element bounds on its upright axes", async () => {
const { agentApiReply } = await tsImport("../../src/report/agent-api.ts", import.meta.url);
const sha = "a".repeat(64);
const map = { screens: [{ id: "scr_home", title: "Home", observationIds: ["obs_1"] }], observations: [{ id: "obs_1", screenshotArtifactId: "art_1" }],
artifacts: [{ id: "art_1", kind: "screenshot", sha256: sha, width: 2853, height: 2007, turned: { width: 669, height: 951 } }],
elements: [{ id: "el_1", screenId: "scr_home", label: "Settings", role: "button", bounds: { x: 600, y: 40, width: 44, height: 30, unit: "points" } }], transitions: [], variants: [], gaps: [] };
const loaded = { kind: "exploration", file: "exploration.json", raw: new Uint8Array(), json: { map }, entries: new Map(), read: async () => undefined };
const reply = await agentApiReply({ pageUrl: "https://example.test/p/token", platforms: ["ios"], expiresAt: null }, loaded, "/screens/scr_home", new URLSearchParams(), "application/json");
const body = JSON.parse(await new Response(reply.body).text());
// Portrait (600, 40) on a 951-point-tall screen shows at (951 - 40 - 30, 600) upright, near the bottom right.
assert.deepEqual(body.elements[0].bounds, { x: 881, y: 600, width: 30, height: 44, unit: "points" });
});
const { assertReplayFits, createReplaySession, parseReplayPlan, replaySetup } = await tsImport("../../src/binary/replay.ts", import.meta.url);
function replayDevice(screens = SCREENS) {
let screen = "home", frames = 0;
const acted = [];
return { acted, options: {
observe: async includeImage => ({ nodes: screens[screen], capturedAt: new Date().toISOString(),
...(includeImage ? { frameId: `replay-${++frames}`, png: PNG, imageSha256: `${frames}`.padStart(64, "b") } : {}) }),
act: async action => {
acted.push(action);
if (action.label === "Missing" || action.label === "Nowhere") throw new Error(`No element labelled ${action.label}`);
if (action.action === "tap" && action.label === "Settings") screen = "settings";
if (action.action === "tap" && action.label === "Dark mode" && screens.settingsDark) screen = "settingsDark";
if (action.action === "back") screen = "home";
},
} };
}
test("a replay plan takes the same steps on each run: platform-only steps are skipped, optional failures are passed over and listed", async () => {
const plan = parseReplayPlan({ steps: [
{ capture: "Home" },
{ action: "tap", label: "Missing", optional: true },
{ action: "tap", label: "Android only", platforms: ["android"] },
{ action: "tap", label: "Settings" }, { capture: "Settings" },
{ action: "tap", label: "Dark mode" }, { capture: "Settings", state: "dark" },
{ action: "back" },
] });
const device = replayDevice();
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
assert.equal(result.stopReason, "finished");
assert.match(result.summary, /Every planned screen was captured\. Optional steps skipped because they failed: step 2 \(tap "Missing"\)\.$/);
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Settings"]);
assert.ok(!device.acted.some(action => action.label === "Android only"));
assert.ok(map.variants.some(variant => variant.name === "dark"), "the dark state is recorded as a variant");
});
test("a failed replay step stops the run, keeps what was captured, and names the screens it did not reach", async () => {
const plan = parseReplayPlan({ steps: [
{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" }, { action: "back" },
{ action: "tap", label: "Nowhere" }, { capture: "Never" }, { capture: "Android screen", platforms: ["android"] },
] });
const device = replayDevice();
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
assert.equal(result.stopReason, "blocked");
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Settings"]);
assert.match(result.summary, /Replay step 5 \(tap "Nowhere"\) failed: .*No element labelled Nowhere.*Not captured: Never$/);
});
test("a blocked step names the screens still ahead before earlier misses, within stop_blocked's limit", async () => {
const plan = parseReplayPlan({ steps: [
{ capture: "Home" },
...Array.from({ length: 20 }, (_, index) => ({ capture: `Earlier miss ${index} ${"x".repeat(60)}` })),
{ action: "tap", label: "Nowhere" },
{ capture: "Ahead" },
] });
const device = replayDevice();
const { result } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options, maxTurns: 100 });
assert.equal(result.stopReason, "blocked");
assert.ok(result.summary.length <= 1000, `${result.summary.length} characters`);
assert.match(result.summary, /Not captured: Ahead, Earlier miss 0 .* and \d+ more$/);
});
test("a destructive or app-leaving tap in a replay plan is refused, as a model's would be", async () => {
const plan = parseReplayPlan({ steps: [
{ capture: "Home" }, { action: "tap", label: "Share", optional: true }, { action: "tap", label: "Delete account" }, { capture: "Gone" },
] });
const device = replayDevice();
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
assert.equal(result.stopReason, "blocked");
assert.match(result.summary, /^Replay step 3 \(tap "Delete account"\) failed: "Delete account" looks destructive.*Not captured: Gone$/);
assert.deepEqual(device.acted, [], "neither tap reaches the device");
assert.deepEqual(map.actionAttempts.map(attempt => [attempt.action.label, attempt.outcome]), [["Share", "blocked"], ["Delete account", "blocked"]]);
});
test("a replay plan is checked before any device starts: unknown, hostile and malformed fields are refused", () => {
const refused = (step, pattern) => assert.throws(() => parseReplayPlan({ steps: [step] }), error => error.code === "REPLAY_PLAN_INVALID" && pattern.test(error.message), JSON.stringify(step));
assert.throws(() => parseReplayPlan({ steps: [] }), /no steps/);
assert.throws(() => parseReplayPlan({ steps: [{ capture: "Home" }], step: [] }), /field "step" it does not take/);
refused({ action: "tap" }, /exactly one of label or identifier/);
refused({ action: "swipe" }, /needs a capture title or an action/);
refused({ action: "constructor" }, /needs a capture title or an action/);
refused({ capture: "Home", platforms: ["web"] }, /platforms must list ios and\/or android/);
// A coordinate tap's fields would turn a label tap into a tap whose risk comes from `target`.
refused({ action: "tap", label: "Delete account", frameId: "frame-1", x: 0.2, y: 0.15, target: "Next" }, /^Step 1 \(tap\) has a field "frameId" it does not take; a tap step takes action, label, identifier, platforms, optional\.$/);
refused({ action: "tap", label: "Next", platform: ["ios"] }, /field "platform"/);
refused({ action: "tap", label: "Next", optinal: true }, /field "optinal"/);
refused({ capture: "Home", optional: true }, /\(capture\) has a field "optional"/);
refused({ capture: "Home", action: "tap", label: "Next" }, /\(capture\) has a field "action"/);
refused({ action: "back", label: "Back" }, /\(back\) has a field "label"/);
refused({ action: "scroll", direction: ["up"] }, /scroll needs a direction/);
refused({ action: "scroll", direction: "sideways" }, /scroll needs a direction/);
refused({ action: "tap", label: "x".repeat(257) }, /label must be 1–256 characters without control characters/);
refused({ action: "tap", identifier: "row\nnext" }, /identifier must be 1–256 characters/);
refused({ action: "fill", label: "Name", text: "two\nlines" }, /fill needs text: one line/);
refused({ action: "type", text: "x".repeat(2001) }, /type needs text/);
refused({ action: "type", text: "{{secret:lowercase}}" }, /invalid secret reference/);
refused({ action: "type", text: "{{secret:OPENAI_API_KEY}}" }, /cannot reference provider/);
refused({ action: "wait", milliseconds: 15_001 }, /between 1 and 15000/);
assert.deepEqual(parseReplayPlan({ steps: [{ capture: "Home" }] }).steps, [{ capture: "Home" }]);
assert.deepEqual(parseReplayPlan({ steps: [{ action: "scroll", direction: "down", optional: true, platforms: ["ios"] }] }).steps,
[{ action: "scroll", direction: "down", platforms: ["ios"], optional: true }]);
});
test("a replay plan with no steps for the platform, or more than the budgets hold, is refused before any device starts", () => {
const limits = { maxScreens: 40, maxActions: 150, maxTurns: 400 };
assert.throws(() => assertReplayFits(parseReplayPlan({ steps: [{ capture: "Home", platforms: ["android"] }, { action: "back", platforms: ["android"] }] }), "ios", limits),
error => error.code === "REPLAY_PLAN_EMPTY" && /No step of the replay plan runs on ios/.test(error.message));
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" }, { action: "back" }, { action: "wait", milliseconds: 100 }] });
assert.doesNotThrow(() => assertReplayFits(plan, "ios", limits));
assert.throws(() => assertReplayFits(plan, "ios", { ...limits, maxScreens: 2, maxActions: 1 }),
error => error.code === "REPLAY_PLAN_OVER_BUDGET" && /2 captures, which need a screen budget of at least 3 \(it is 2\) and 2 actions, more than the action budget of 1/.test(error.message)
&& /--max-screens 3 --max-actions 2/.test(error.nextAction));
assert.throws(() => assertReplayFits(plan, "ios", { ...limits, maxTurns: 8 }), /takes 9 tool calls, more than the 8/);
});
test("a replay that runs out of budget names the planned screens it did not reach in the gap", async () => {
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" }, { action: "back" }, { capture: "Later" }] });
const device = replayDevice();
const session = createReplaySession(plan, "ios");
const { result, map } = await harness([], { session, budgetNote: session.budgetNote, ...device.options, maxScreens: 2 });
assert.equal(result.stopReason, "budget-screens");
assert.match(map.gaps.at(-1).message, /^The exploration stopped at its screens budget with \d+ controls unexplored\. Not captured: Settings, Later\.$/);
});
test("a capture of a screen the plan already captured under another title is reported and does not rename it", async () => {
const plan = parseReplayPlan({ steps: [
{ capture: "Home" }, { capture: "Start" }, { capture: "home" }, { action: "tap", label: "Settings" }, { action: "back" },
] });
const device = replayDevice();
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
assert.equal(result.stopReason, "finished");
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Home → Settings"]);
assert.match(result.summary, /Not captured: Start \(looked the same as Home\)\. Also reached, not in the plan: 1 unnamed screen\.$/);
});
test("a planned state that the explorer merges into an earlier screen is neither reported as unplanned nor left in the map", async () => {
// The README's own example: Dark mode changes the tree, and the capture titled Settings folds that new screen into Settings.
const screens = { ...SCREENS, settingsDark: [...SCREENS.settings, { type: "Button", label: "Theme options" }] };
const plan = parseReplayPlan({ steps: [
{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" },
{ action: "tap", label: "Dark mode" }, { capture: "Settings", state: "dark" },
] });
const device = replayDevice(screens);
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
assert.equal(result.stopReason, "finished");
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Settings"]);
const settings = map.screens.find(screen => screen.title === "Settings");
assert.ok(map.variants.some(variant => variant.screenId === settings.id && variant.name === "dark"));
assert.equal(result.summary, "Replayed 5 planned steps on ios. Every planned screen was captured.");
});
test("a long replay summary is kept within finish's limit, the most useful names first", async () => {
const titles = Array.from({ length: 60 }, (_, index) => `Planned screen ${String(index).padStart(2, "0")} ${"y".repeat(60)}`);
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, ...titles.map(title => ({ capture: title }))] });
const device = replayDevice();
const { result } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options, maxTurns: 200 });
assert.equal(result.stopReason, "finished");
assert.ok(result.summary.length <= 4000, `${result.summary.length} characters`);
assert.match(result.summary, /^Replayed 61 planned steps on ios\. Not captured: Planned screen 00 y+ \(looked the same as Home\), .* and \d+ more\.$/);
});
test("a refused finish ends the replay with stop_blocked instead of a turn budget, and a refused stop is an error", async () => {
const plan = parseReplayPlan({ steps: [{ action: "back" }] });
const calls = [];
const request = refuse => ({ signal: new AbortController().signal, callTool: async (name, input) => {
calls.push({ name, input });
const value = name === "get_progress" ? { captured: [] } : refuse.includes(name) ? { ok: false, code: "AGENT_ACTION_INVALID", message: `${name} refused` } : { ok: true };
return { content: [{ type: "text", text: JSON.stringify(value) }], ...(refuse.includes(name) ? { isError: true } : {}) };
} });
await createReplaySession(plan, "ios").run(request(["finish"]));
assert.deepEqual(calls.map(call => call.name), ["act", "get_progress", "finish", "stop_blocked"]);
assert.match(calls.at(-1).input.reason, /^The replay could not finish \(finish refused\)\. Replayed 1 planned steps on ios\./);
await assert.rejects(createReplaySession(plan, "ios").run(request(["finish", "stop_blocked"])), { code: "REPLAY_STOP_REFUSED" });
});
test("a replay plan's {{secret:NAME}} inputs are read before any device starts and typed for the plan", async () => {
const { runBinaryExploration, runSessionExploration } = await tsImport("../../src/binary/run.ts", import.meta.url);
const { createExplorationInputs } = await tsImport("../../src/config/exploration-inputs.ts", import.meta.url);
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, { action: "fill", label: "Password", text: "{{secret:APP_TEST_PASSWORD}}" }] });
const output = await mkdtemp(join(tmpdir(), "explore-replay-"));
for (const run of [runBinaryExploration, runSessionExploration]) {
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: plan, environment: {} }),
error => error.code === "EXPLORATION_SECRET_MISSING" && /APP_TEST_PASSWORD/.test(error.message));
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: plan, environment: {}, provider: "openai", hints: "Look around" }),
error => error.code === "REPLAY_OPTIONS_IGNORED" && /--provider and --hints would do nothing/.test(error.message));
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: { steps: [{ action: "tap", label: "Next", optinal: true }] }, environment: {} }), { code: "REPLAY_PLAN_INVALID" });
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: parseReplayPlan({ steps: [{ action: "open", url: "https://example.com" }] }), environment: {} }),
error => error.code === "REPLAY_PLAN_INVALID" && /^Step 1: https: URLs leave the app/.test(error.message));
}
const device = replayDevice();
const inputs = createExplorationInputs({ setup: replaySetup(plan), environment: { APP_TEST_PASSWORD: "hunter2-test" } });
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), inputs, ...device.options });
assert.equal(result.stopReason, "finished");
assert.deepEqual(device.acted, [{ action: "fill", label: "Password", text: "hunter2-test" }]);
assert.equal(map.actionAttempts[0].action.input.value, "{{secret:APP_TEST_PASSWORD}}");
assert.ok(!JSON.stringify(map).includes("hunter2-test"));
});
+36 -5
View File
@@ -29,25 +29,25 @@ async function simulatorApp(root) {
}
/** simctl and agent-device on a Mac with one iPhone 17 Pro runtime; the app shows a single screen. */
function localRunner() {
function localRunner(device = { name: "iPhone 17 Pro", identifier: DEVICE_TYPE }, screen = { nodes: [{ type: "StaticText", label: "Welcome" }, { type: "Button", label: "Settings" }], png: PNG }) {
const requests = [];
const ok = data => ({ stdout: JSON.stringify({ success: true, data }), stderr: "" });
const runner = request => {
requests.push(request);
const [first, second, third] = request.args;
if (request.command === "xcrun") {
if (second === "list" && third === "devicetypes") return { stdout: JSON.stringify({ devicetypes: [{ name: "iPhone 17 Pro", identifier: DEVICE_TYPE }] }), stderr: "" };
if (second === "list" && third === "devicetypes") return { stdout: JSON.stringify({ devicetypes: [device] }), stderr: "" };
if (second === "list" && third === "runtimes") return { stdout: JSON.stringify({ runtimes: [{ isAvailable: true, platform: "iOS", identifier: "com.apple.CoreSimulator.SimRuntime.iOS-27-0",
version: "27.0", supportedDeviceTypes: [{ identifier: DEVICE_TYPE }] }] }), stderr: "" };
version: "27.0", supportedDeviceTypes: [{ identifier: device.identifier }] }] }), stderr: "" };
if (second === "create") return { stdout: `${UDID}\n`, stderr: "" };
return { stdout: "", stderr: "" };
}
if (first === "--version") return { stdout: `${controllerVersion("agent-device")}\n`, stderr: "" };
if (first === "open") return ok({ session: CONTROLLER_SESSION, appBundleId: "dev.example.app", device: { platform: "ios", target: "mobile", name: "Explore iPhone 17 Pro", id: UDID },
identifiers: { session: CONTROLLER_SESSION, appBundleId: "dev.example.app", deviceName: "Explore iPhone 17 Pro", deviceId: UDID } });
if (first === "snapshot") return ok({ nodes: [{ type: "StaticText", label: "Welcome" }, { type: "Button", label: "Settings" }] });
if (first === "snapshot") return ok({ nodes: screen.nodes });
if (first === "appstate") return ok({ platform: "ios", appBundleId: "dev.example.app", state: "runningForeground", surface: "app" });
if (first === "screenshot") { writeFileSync(request.args[1], PNG); return ok({}); }
if (first === "screenshot") { writeFileSync(request.args[1], screen.png); return ok({}); }
return ok({});
};
return { requests, runner };
@@ -85,3 +85,34 @@ test("explore:binary --simulator local runs on a simulator it creates, with no E
assert.equal(result.session.id, `local-${UDID}`);
assert.deepEqual(await readdir(join(root, ".cache", "expo-explore", "local-captures")), [], "nothing is left for a sweep");
});
test("an opened iPhone Duo's screenshots are stored upright, and the run records the landscape they show", async t => {
const root = await mkdtemp(join(tmpdir(), "explore-binary-duo-"));
t.after(() => rm(root, { recursive: true, force: true }));
const { createRequire } = await import("node:module");
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
// The inner panel on its portrait axes at 1x, as local agent-device returns it: the tree's root is 669×951 points.
const raw = new Png({ width: 669, height: 951 });
for (let offset = 0; offset < raw.data.length; offset += 4) raw.data.set(offset < 669 * 4 ? [255, 0, 0, 255] : [255, 255, 255, 255], offset);
const nodes = [{ type: "Application", label: "Example", rect: { x: 0, y: 0, width: 669, height: 951 } }, { type: "Button", label: "Settings", rect: { x: 20, y: 40, width: 80, height: 44 } }];
const { runner } = localRunner({ name: "iPhone Duo", identifier: "com.apple.CoreSimulator.SimDeviceType.iPhone-Duo" }, { nodes, png: Png.sync.write(raw) });
const session = { provider: "anthropic", model: "test", async preflight() {}, async run(request) {
const observed = JSON.parse((await request.callTool("observe", { includeImage: true }, request.signal)).content[0].text);
await request.callTool("capture", { frameId: observed.frameId, title: "Home" }, request.signal);
await request.callTool("finish", { summary: "One." }, request.signal).catch(() => {});
return { stopReason: "done" };
} };
const result = await runBinaryExploration({ archive: await simulatorApp(root), output: join(root, "out"), simulator: "local", device: "iPhone Duo", provider: "anthropic", runner,
createSession: () => session, environment: { PATH: process.env.PATH, HOME: root, ANTHROPIC_API_KEY: "key" } });
const { readFile } = await import("node:fs/promises");
const map = JSON.parse(await readFile(result.mapPath, "utf8"));
const artifact = map.artifacts.find(item => item.kind === "screenshot");
assert.deepEqual([artifact.width, artifact.height, artifact.turned], [951, 669, { width: 669, height: 951 }]);
const stored = Png.sync.read(await readFile(join(result.outputDirectory, artifact.path)));
assert.deepEqual([stored.width, stored.height], [951, 669]);
// The panel's top row (red) is the right-hand column once turned.
assert.deepEqual([...stored.data.subarray((951 - 1) * 4, 951 * 4)], [255, 0, 0, 255]);
assert.deepEqual([...stored.data.subarray(0, 4)], [255, 255, 255, 255]);
assert.equal(map.run.environment.orientation, "landscape");
assert.equal(result.runtime.orientation, "landscape");
});
+56
View File
@@ -69,6 +69,18 @@ const unreadableSurface = () => json({ success: false, error: { code: 'COMMAND_F
details: { hint: 'com.apple.SafariViewService hosts the surface presented over the app, and this capture targeted that surface. No snapshot backend could read this screen (XCTest-backed snapshot tiers were deferred after recent slow accessibility work on this screen).' } } });
/**
* The iOS 26 keyboard as agent-device reports it: the Keyboard element with its keys beneath it, and the dock's Emoji and
* Dictate buttons, which its host bridge names by their view class (brentvatne/euxy#172, review run 39362642343).
*/
const softKeyboard = [
{ index: 900, type: 'Application', rect: { x: 0, y: 0, width: 402, height: 874 }, visible: true },
{ index: 902, parentIndex: 900, type: 'Keyboard', rect: { x: 0, y: 583, width: 402, height: 233 }, visible: true },
{ index: 903, parentIndex: 902, type: 'Key', label: 'Q', rect: { x: 5, y: 590, width: 39, height: 54 }, visible: true },
{ index: 905, parentIndex: 900, type: 'Button', role: 'UIKeyboardDockItemButton', label: 'Emoji', rect: { x: 8, y: 806, width: 68, height: 68 }, visible: true },
{ index: 906, parentIndex: 900, type: 'Button', role: 'UIKeyboardDockItemButton', label: 'Dictate', identifier: 'dictation', rect: { x: 325, y: 806, width: 68, height: 68 }, visible: true },
];
function agentFixture(actions, observedRequests = []) {
return { instructions: 'Capture each selected route in its normal loaded state.', contexts: {},
provider: { provider: 'anthropic', model: 'test-model', preflight: async () => {}, run: async request => {
@@ -117,6 +129,8 @@ async function fixture(t, settings = {}) {
let enteredName = false; let scrolledToContinue = false; let finishHidden = false; let wedgeCleared = false; let deviceLost = false; let relaunchedAfterForeign = false; let serverStarts = 0; let unreadableLeft = settings.unreadableAfterFold ?? 0;
// startsErrored / startsUnnamed: how many session starts EAS fails, after naming the session (it ERRORS) or before.
let starts = 0; let currentSession = 'session-1'; const erroredSessions = new Set(); let foldUnverifiedLeft = settings.foldUnverified ?? 0;
// focusedField: the screen at `path` focuses its text field, and the keyboard is up from the `keyboardFrom`th read of it.
let focusedReads = 0;
// blackAfterFold: the first so many folds of an EAS agent-device session light the panel but draw nothing.
let folds = 0;
const deviceName = settings.android ? 'Pixel 7' : 'iPhone 17 Pro';
@@ -369,6 +383,8 @@ async function fixture(t, settings = {}) {
] : []),
...(settings.statefulSetup && current === '/transient' ? [{ type: 'Button', label: 'Finish', ref: 'e103', enabled: true, visible: !finishHidden }] : []),
...(alert ? [{ type: 'Alert', label: alert.message }, ...alert.items.map((label, index) => ({ type: 'Button', label, ref: `e${index + 1}`, enabled: true, visible: true }))] : []),
...(settings.focusedField?.path === current ? [{ type: 'TextField', label: 'Pattern name', focused: true, visible: true, rect: { x: 23, y: 386, width: 355, height: 50 } },
...(++focusedReads >= (settings.focusedField.keyboardFrom ?? Infinity) ? softKeyboard : [])] : []),
...(settings.missingObserver || settings.missingObserverAt === current ? [] : [observer]),
// A second active destination is ambiguous; the host AX bridge's identical copy is not.
...(settings.duplicateObserver ? [{ ...observer, label: JSON.stringify({ provider: 'expo-router', pathname: '/other', segments: ['other'], parameters: {} }) }] : []),
@@ -1181,6 +1197,46 @@ test('every app link settles before the screen is read', async t => {
assert.ok(opened >= 0 && settled > opened && settled < read, 'the settle runs between the link and the first read');
});
const keyboardLog = async f => (await readFile(join(f.options.mapDirectory, 'capture-diagnostics', 'keyboard.jsonl'), 'utf8'))
.trim().split('\n').map(line => JSON.parse(line));
test('a screen whose text field has the focus is captured once its keyboard is up and still, and the keyboard is not one of its elements', async t => {
// euxy#172 (review run 39362642343): the rename sheet focuses its field, and the keyboard rises after the sheet settles.
const f = await fixture(t, { routes: [route('rename')], focusedField: { path: '/rename', keyboardFrom: 2 } });
const result = await runCiMap(f.options);
assert.equal(result.outcomes[f.ids.rename].status, 'captured');
assert.equal(result.outcomes[f.ids.rename].attempts, 1);
assert.deepEqual((await keyboardLog(f)).map(line => [line.route, line.outcome, line.reads]), [[f.ids.rename, 'shown', 2]],
'read until the keyboard was up in the same place twice');
const calls = f.requests.filter(request => request.command === 'fake-controller').map(request => request.args[0]);
const shot = calls.lastIndexOf('screenshot');
assert.deepEqual(calls.slice(shot - 3, shot), ['snapshot', 'snapshot', 'snapshot'], 'the destination read, then the keyboard reads, then the screenshot');
const map = await readExportedMap(f.options.outputDirectory);
const labels = map.observations.flatMap(observation => observation.elements.map(element => element.label));
assert.ok(labels.includes('Pattern name'));
for (const label of ['Q', 'Emoji', 'Dictate']) assert.equal(labels.includes(label), false, `${label} is the keyboard's, not the app's`);
});
test('a focused field whose keyboard never rises is captured after about a second and at least two reads, without it', async t => {
const f = await fixture(t, { routes: [route('rename')], focusedField: { path: '/rename' } });
const result = await runCiMap(f.options);
assert.equal(result.outcomes[f.ids.rename].status, 'captured');
const [line] = await keyboardLog(f);
assert.equal(line.outcome, 'absent');
assert.ok(line.reads >= 2 && line.reads <= 6 && line.elapsedMs >= 1000, `gave up after ${line.reads} reads in ${line.elapsedMs} ms`);
assert.equal(f.operations().includes('keyboard'), false, 'the keyboard is never dismissed or summoned');
// A screen without a focused field is not read again.
const plain = await fixture(t, { routes: [route('home')] });
await runCiMap(plain.options);
assert.equal(existsSync(join(plain.options.mapDirectory, 'capture-diagnostics', 'keyboard.jsonl')), false);
});
test('a keyboard already up when the screen is read is captured once a later read finds it in the same place', async t => {
const f = await fixture(t, { routes: [route('rename')], focusedField: { path: '/rename', keyboardFrom: 1 } });
assert.equal((await runCiMap(f.options)).outcomes[f.ids.rename].status, 'captured');
assert.deepEqual((await keyboardLog(f)).map(line => [line.outcome, line.reads]), [['shown', 1]]);
});
test('an application Open button is never an implicit confirmation action', async t => {
const f = await fixture(t, { routes: [route('home')], appOpenButton: true });
const result = await runCiMap(f.options);
+12
View File
@@ -331,3 +331,15 @@ test("compare CLI emits a comparison, review, and catalog that reload as a works
));
assert.deepEqual(shared.baselineScreenIds, ["shared-home"]);
});
test("fingerprints from different Explore versions are not comparable: the match needs review instead of reading as changed", async () => {
// A map saved by Explore 0.20.x hashes the tree with the keyboard and the focus in it, unprefixed; a2 hashes leave them out.
const baseline = makeMap("main-eeeeeee", [{ id: "home", route: "/", acc: "0".repeat(64) }]);
const current = makeMap("pr-3-fffffff", [{ id: "home", route: "/", acc: `a2:${"1".repeat(64)}` }]);
const { comparison, review } = generateComparison(baseline, current);
await validateComparisonDocument(comparison, baseline, current);
const home = review.entries.find((entry) => entry.entryId === "match-home");
assert.equal(home.needsReview, true);
assert.match(entryById(comparison, "match-home").rationale, /different Explore versions/);
assert.ok(home.notes.some((note) => /compare the screenshots/.test(note)));
});
+45
View File
@@ -823,3 +823,48 @@ test("a capture's snapshot backend is read from its saved controller response",
assert.equal(snapshotBackend(directory, map("artifacts/accessibility/missing.json"), "o"), undefined);
assert.equal(snapshotBackend(directory, map("../outside.json"), "o"), undefined);
});
/**
* Tempo on the closed iPhone Duo in brentvatne/euxy#172 (review run 39362642343): a 1398 × 2034 screenshot of a
* 466 × 678 pt screen, and a footnote whose text changed below the fold of its scrolling sheet, so the screenshots match.
* The published review carries no accessibility tree, so the footnote's real bounds are unknown: these tests pin the
* rule (more than half of the changed element past the screen's edge), not where euxy's footnote was.
*/
function tempoSide(sideName, footnote, footnoteBounds, image = { width: 1398, height: 2034 }) {
const evidence = side(sideName, [
element({ elementId: "app", role: "other", label: "euxy", actions: [], actionable: false, bounds: { x: 0, y: 0, width: 466, height: 678, unit: "points" } }),
element({ elementId: "title", role: "text", label: "Base Resolution", actions: [], actionable: false, bounds: { x: 26, y: 560, width: 300, height: 24, unit: "points" } }),
element({ elementId: `${sideName}-footnote`, role: "text", label: footnote, actions: [], actionable: false, bounds: footnoteBounds }),
]);
if (image) {
evidence.map.artifacts.push({ id: `${sideName}-shot`, kind: "screenshot", path: `${sideName}.png`, sha256: "0".repeat(64), mimeType: "image/png", ...image, createdAt: "2026-09-01T01:00:00.000Z" });
evidence.map.observations[0].screenshotArtifactId = `${sideName}-shot`;
}
return evidence;
}
const NEW_LANES = "Changes apply live while playing. Base resolution sets the step grid for new lanes.";
const ADD_NEXT = "Changes apply live while playing. Base resolution sets the step grid for lanes you add next.";
// Its top 12 pt on the screen, 28 pt below the bottom edge: 70% past it.
const mostlyBelow = { x: 26, y: 666, width: 414, height: 40, unit: "points" };
const offScreenOf = (bounds, assessment = visual("unchanged"), image) => compareSurfaceEvidence({ impact: impact(),
base: tempoSide("base", NEW_LANES, bounds, image), head: tempoSide("head", ADD_NEXT, bounds, image), visualAssessment: assessment }).offScreen;
test("matching screenshots of a change whose elements lie mostly past the screen's edge are off-screen", () => {
assert.equal(offScreenOf(mostlyBelow), true);
// A turned frame (an opened iPhone Duo shown upright) keeps the tree on the portrait axes: the screen is still found.
assert.equal(offScreenOf(mostlyBelow, visual("unchanged"), { width: 2034, height: 1398 }), true);
});
test("a change on the screen, partly on it, with differing screenshots, or with no screen to measure against is never off-screen", () => {
const cases = {
"an on-screen text change": [{ x: 26, y: 600, width: 414, height: 40, unit: "points" }],
// Its top 28 pt on the screen, 12 pt below: mostly visible, so matching pixels prove nothing about it.
"a footnote mostly on the screen": [{ x: 26, y: 650, width: 414, height: 40, unit: "points" }],
// A control nudged 2 pt past the left edge, otherwise wholly visible.
"a control just crossing an edge": [{ x: -2, y: 600, width: 300, height: 40, unit: "points" }],
"screenshots that differ": [mostlyBelow, visual("modified")],
"no screenshot size": [mostlyBelow, visual("unchanged"), null],
"a changed element without bounds": [undefined],
};
for (const [name, args] of Object.entries(cases)) assert.equal(offScreenOf(...args), undefined, name);
});
+201
View File
@@ -0,0 +1,201 @@
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import test from "node:test";
import { tsImport } from "tsx/esm/api";
const { snapshotFromTree } = await tsImport("../../src/capture/argent-driver.ts", import.meta.url);
const { focusedTextInput, keyboardBounds, keyboardNodes, keyboardSettle } = await tsImport("../../src/capture/keyboard.ts", import.meta.url);
const { parseControllerOutput } = await tsImport("../../src/capture/controller-output.ts", import.meta.url);
const { normalizeSnapshot } = await tsImport("../../src/map/store.ts", import.meta.url);
const fixture = name => JSON.parse(readFileSync(new URL(`../fixtures/argent-0.29/${name}`, import.meta.url), "utf8"));
// euxy's rename sheet on iOS 26 under Argent 0.29, its field focused and the keyboard up: the head of brentvatne/euxy#172
// (review run 39362642343). Its root holds three windows: the app's, the input view placeholder's, and the keyboard's.
const RENAME_KEYBOARD_UP = fixture("fixture-ios-ui-tree-rename.json");
// The same sheet a moment earlier, as the base was captured: the field focused and its cursor showing, no keyboard yet.
const RENAME_KEYBOARD_DOWN = { ...RENAME_KEYBOARD_UP, keyboardVisible: false,
roots: [{ ...RENAME_KEYBOARD_UP.roots[0], children: RENAME_KEYBOARD_UP.roots[0].children.slice(0, 1) }] };
const ANDROID_RENAME = fixture("fixture-android-ui-tree-rename.json");
const nodesOf = tree => snapshotFromTree(tree, "ios", 1).nodes;
const labelsOf = nodes => parseControllerOutput("agent-device", JSON.stringify({ success: true, data: { nodes } })).elements.map(element => element.label);
const nameOf = node => node.label ?? node.identifier;
const rect = (x, y, width, height) => ({ x, y, width, height });
test("the iOS keyboard Argent reports is its own windows: the input view placeholder's and the keyboard's, Emoji and Dictate included", () => {
const nodes = nodesOf(RENAME_KEYBOARD_UP);
assert.deepEqual(keyboardBounds(nodes), rect(0, 539, 402, 335), "the input view and prediction bar down to the bottom edge");
const owned = [...keyboardNodes(nodes)];
const names = owned.map(nameOf).filter(Boolean);
for (const name of ["inputView", "SystemInputAssistantView", "Typing Predictions", "Q", "shift", "Emoji", "Dictate"]) assert.ok(names.includes(name), name);
assert.ok(owned.every(node => node.systemKeyboard === true), "only nodes of the keyboard's windows");
// Every element of the app's window stays, those behind the keyboard's band among them.
const app = nodesOf(RENAME_KEYBOARD_DOWN).map(nameOf).filter(Boolean);
for (const name of app) assert.ok(nodes.some(node => nameOf(node) === name && !keyboardNodes(nodes).has(node)), name);
assert.ok(app.includes("AdditionalDimmingOverlay") && app.includes("Tab Bar") && app.includes("Clear name"));
});
test("a keyboard up on one side only leaves the elements and the accessibility fingerprint unchanged", () => {
// In euxy#172 the review led with "Added button “Dictate”", “Emoji” and “inputView” over the label change.
const up = nodesOf(RENAME_KEYBOARD_UP), down = nodesOf(RENAME_KEYBOARD_DOWN);
assert.deepEqual(labelsOf(up), labelsOf(down));
for (const name of ["Dictate", "Emoji", "inputView", "Q"]) assert.equal(labelsOf(up).includes(name), false, name);
// The bridge numbers nodes in order, so the app's own nodes keep their indices when the keyboard's windows follow them.
const snapshot = nodes => JSON.stringify({ success: true, data: { nodes: nodes.map(({ ref: _ref, ...node }) => node) } });
assert.equal(normalizeSnapshot(snapshot(up)), normalizeSnapshot(snapshot(down)));
});
test("the text field with the focus is passed on from Argent", () => {
const up = nodesOf(RENAME_KEYBOARD_UP), down = nodesOf(RENAME_KEYBOARD_DOWN);
assert.equal(up.filter(node => node.focused === true).map(node => node.type).join(), "TextField");
assert.equal(focusedTextInput(up), true);
assert.equal(focusedTextInput(down), true);
assert.equal(keyboardBounds(down), undefined);
assert.equal(focusedTextInput(nodesOf(fixture("fixture-ios-ui-tree-home.json"))), false);
});
test("an element the app identifies as inputView in its own window is the app's", () => {
// A chat composer with testID="inputView", no keyboard up.
const app = RENAME_KEYBOARD_DOWN.roots[0].children[0];
const composer = { role: "AXGroup", type: "Other", identifier: "inputView", frame: rect(0, 0.8, 1, 0.1),
children: [{ role: "AXButton", type: "Button", label: "Send", frame: rect(0.8, 0.82, 0.15, 0.05), children: [] }] };
const tree = { ...RENAME_KEYBOARD_DOWN, roots: [{ ...RENAME_KEYBOARD_DOWN.roots[0], children: [{ ...app, children: [...app.children, composer] }] }] };
const nodes = nodesOf(tree);
assert.equal(keyboardNodes(nodes).size, 0);
assert.equal(keyboardBounds(nodes), undefined);
assert.ok(labelsOf(nodes).includes("inputView") && labelsOf(nodes).includes("Send"));
});
test("a second window of the app's own, its testID inputView included, is the app's; the keyboard's windows need a Keyboard", () => {
// The app's second UIWindow (a toast or a floating composer) with a view identified as inputView and a Save button.
const window = { role: "AXGroup", type: "Application", frame: rect(0, 0, 1, 1), children: [
{ role: "AXGroup", type: "Other", identifier: "inputView", frame: rect(0, 0.8, 1, 0.1),
children: [{ role: "AXButton", type: "Button", label: "Save", frame: rect(0.8, 0.82, 0.15, 0.05), children: [] }] }] };
const [app, placeholder, keyboard] = RENAME_KEYBOARD_UP.roots[0].children;
const withWindows = windows => nodesOf({ ...RENAME_KEYBOARD_UP, roots: [{ ...RENAME_KEYBOARD_UP.roots[0], children: [app, ...windows] }] });
for (const windows of [[window], [window, keyboard], [window, placeholder, keyboard]]) {
const labels = labelsOf(withWindows(windows));
assert.ok(labels.includes("Save") && labels.includes("inputView"), `kept beside ${windows.length - 1} keyboard windows`);
}
// Without the keyboard's own window, an input view placeholder window proves nothing.
assert.ok(labelsOf(withWindows([placeholder])).includes("inputView"));
assert.equal(keyboardNodes(withWindows([placeholder])).size, 0);
// With it, the placeholder is the keyboard's too, as in the real tree.
assert.equal(labelsOf(withWindows([placeholder, keyboard])).includes("inputView"), false);
});
test("agent-device's keyboard is its Keyboard element and what is under it, and its host bridge's keyboard view classes", () => {
// XCTest: the Keyboard directly under the Application, beside the app's own controls in the same band.
const xctest = [
{ index: 0, type: "Application", label: "Example", rect: rect(0, 0, 402, 874) },
{ index: 1, parentIndex: 0, type: "TextField", label: "Pattern name", focused: true, rect: rect(23, 386, 355, 50) },
{ index: 2, parentIndex: 0, type: "Keyboard", rect: rect(0, 583, 402, 233) },
{ index: 3, parentIndex: 2, type: "Key", label: "Q", rect: rect(5, 590, 39, 54) },
{ index: 4, parentIndex: 0, type: "Button", label: "Home tab", rect: rect(0, 800, 134, 74) },
{ index: 5, parentIndex: 0, type: "Other", label: "Formatting toolbar", rect: rect(0, 540, 402, 43) },
{ index: 6, parentIndex: 0, type: "Button", label: "Emoji", rect: rect(8, 300, 68, 40) },
];
assert.deepEqual(keyboardBounds(xctest), rect(0, 583, 402, 233));
assert.deepEqual(labelsOf(xctest), ["Example", "Pattern name", "Home tab", "Formatting toolbar", "Emoji"]);
// The host bridge names the dock's buttons by class, wherever they sit.
const hostBridge = [
{ type: "TextField", role: "RCTUITextField", label: "Pattern name", focused: true, rect: rect(23, 386, 355, 50) },
{ type: "Button", role: "UIKeyboardDockItemButton", label: "Dictate", rect: rect(325, 806, 68, 68) },
{ type: "Other", role: "UIAccessibilityElementKBKey", label: "Q", rect: rect(5, 590, 39, 54) },
{ type: "Button", role: "RCTViewComponentView", label: "Profile tab", rect: rect(268, 800, 134, 74) },
];
assert.ok(keyboardBounds(hostBridge));
assert.deepEqual(labelsOf(hostBridge), ["Pattern name", "Profile tab"]);
});
test("nothing without a parent link is taken for the keyboard's, and keys alone are no keyboard", () => {
// agent-device's flat shape: no parentIndex. The Keyboard element goes; nothing can be proven to be under it.
const flat = [
{ type: "Keyboard", rect: rect(0, 583, 402, 233) },
{ type: "Key", label: "Q", rect: rect(5, 590, 39, 54) },
{ type: "Button", label: "Profile tab", rect: rect(268, 800, 134, 74) },
];
assert.deepEqual(labelsOf(flat), ["Q", "Profile tab"]);
// An app's own PIN pad (accessibilityRole="keyboardkey") and its footer action, with no system keyboard up.
const pinPad = [
{ index: 0, type: "Application", label: "Example", rect: rect(0, 0, 402, 874) },
{ index: 1, parentIndex: 0, type: "SecureTextField", label: "PIN", focused: true, rect: rect(23, 200, 355, 50) },
...["1", "2", "3"].map((digit, offset) => ({ index: 2 + offset, parentIndex: 0, type: "Key", label: digit, rect: rect(offset * 134, 600, 134, 70) })),
{ index: 5, parentIndex: 0, type: "Button", label: "App footer action", rect: rect(0, 800, 402, 74) },
];
assert.equal(keyboardNodes(pinPad).size, 0);
assert.equal(keyboardBounds(pinPad), undefined);
assert.deepEqual(labelsOf(pinPad), ["Example", "PIN", "1", "2", "3", "App footer action"]);
});
test("in landscape the keyboard is told apart the same way, the app's bottom bar kept", () => {
const landscape = [
{ index: 0, type: "Application", label: "Example", rect: rect(0, 0, 874, 402) },
{ index: 1, parentIndex: 0, type: "TextField", label: "Search", focused: true, rect: rect(100, 60, 674, 40) },
{ index: 2, parentIndex: 0, type: "Keyboard", rect: rect(0, 230, 874, 172) },
{ index: 3, parentIndex: 2, type: "Key", label: "Q", rect: rect(60, 236, 70, 38) },
{ index: 4, parentIndex: 0, type: "TabBar", label: "Tab Bar", rect: rect(0, 352, 874, 50) },
];
assert.deepEqual(keyboardBounds(landscape), rect(0, 230, 874, 172));
assert.deepEqual(labelsOf(landscape), ["Example", "Search", "Tab Bar"]);
});
test("an Android input method's window is the keyboard under Argent, AOSP's included, and Argent says when one shows", () => {
const ime = pkg => ({ role: "FrameLayout", type: "android.widget.FrameLayout", bundleId: pkg, frame: rect(0, 0.6, 1, 0.4),
children: [{ role: "Button", type: "android.widget.Button", contentDescription: "q", frame: rect(0.01, 0.65, 0.09, 0.06), children: [] }] });
const before = snapshotFromTree(ANDROID_RENAME, "android", 1, "dev.brent.euxy").nodes;
assert.equal(focusedTextInput(before), true, "Argent marks the focused EditText");
assert.equal(keyboardBounds(before), undefined);
for (const pkg of ["com.google.android.inputmethod.latin", "com.android.inputmethod.latin"]) {
const after = snapshotFromTree({ ...ANDROID_RENAME, keyboardVisible: true, roots: [...ANDROID_RENAME.roots, ime(pkg)] }, "android", 1, "dev.brent.euxy").nodes;
assert.deepEqual(keyboardBounds(after), rect(0, 960, 720, 640), pkg);
assert.deepEqual(labelsOf(after), labelsOf(before), pkg);
}
});
test("the wait reads until the keyboard is in one place twice, at least twice, at most six times", () => {
const field = { type: "TextField", label: "Pattern name", focused: true, rect: rect(23, 386, 355, 50) };
const keyboardAt = y => [field, { index: 9, type: "Keyboard", rect: rect(0, y, 402, 874 - y) }];
// Already up when the screen was read: one more read must find it in the same place.
const up = keyboardSettle({ nodes: keyboardAt(583) });
assert.equal(up.more(0), true, "a keyboard already up may still be rising");
up.read({ nodes: keyboardAt(583) });
assert.deepEqual([up.outcome, up.reads, up.more(250)], ["shown", 1, false]);
// Caught rising: it moves between reads, then holds.
const rising = keyboardSettle({ nodes: [field] });
for (const y of [700, 583, 583]) rising.read({ nodes: keyboardAt(y) });
assert.deepEqual([rising.outcome, rising.reads], ["shown", 3]);
// A slow controller: the second read is owed even after a second has passed.
const slow = keyboardSettle({ nodes: [field] });
slow.read({ nodes: [field] });
assert.equal(slow.more(2_500), true);
slow.read({ nodes: [field] });
assert.deepEqual([slow.outcome, slow.more(5_000)], ["absent", false], "no keyboard came: captured without one");
// A slow first read without a keyboard, then the keyboard first seen still moving: read on until it holds.
const late = keyboardSettle({ nodes: [field] });
late.read({ nodes: [field] });
late.read({ nodes: keyboardAt(700) });
assert.deepEqual([late.outcome, late.more(2_500)], ["moving", true], "a moving keyboard is not given up on after a second");
late.read({ nodes: keyboardAt(583) });
late.read({ nodes: keyboardAt(583) });
assert.deepEqual([late.outcome, late.reads], ["shown", 4]);
// But not past four seconds, however few reads that was.
const stalled = keyboardSettle({ nodes: [field] });
stalled.read({ nodes: [field] });
stalled.read({ nodes: keyboardAt(700) });
assert.equal(stalled.more(4_000), false);
// Six reads at most, however fast they come.
const busy = keyboardSettle({ nodes: [field] });
for (let read = 0; busy.more(0); read++) busy.read({ nodes: keyboardAt(read % 2 ? 583 : 700) });
assert.deepEqual([busy.outcome, busy.reads], ["moving", 6]);
// The field let go and no keyboard shows: nothing more to wait for.
const released = keyboardSettle({ nodes: [field] });
released.read({ nodes: [{ ...field, focused: false }] });
assert.deepEqual([released.outcome, released.more(0)], ["unfocused", false]);
// Argent on Android says a keyboard shows without placing it: it shows twice running.
const android = keyboardSettle({ nodes: [field] });
android.read({ nodes: [field], keyboardVisible: true });
android.read({ nodes: [field], keyboardVisible: true });
assert.equal(android.outcome, "shown");
assert.equal(keyboardSettle({ nodes: [{ ...field, focused: false }] }), undefined, "a screen without a focused field is not waited on");
});
+74
View File
@@ -679,6 +679,80 @@ test('a config without targets writes the same records and comment as before tar
assert.deepEqual(Object.keys(readCommentRecords(after).current), ['ios', 'android']);
});
/**
* brentvatne/euxy#172, review run 39362642343: five targets, two strings changed. Tempo's footnote sits low in a
* scrolling sheet. On the closed iPhone Duo the accessibility tree has the new text but the screenshots match: the
* footnote runs past the bottom of the screen (`offScreen`, from the elements' bounds). On the half-open and opened Duo
* neither the tree nor the pixels have it. Android captures no dark appearance.
*/
const euxyTargets = [{ id: 'iphone', platform: 'ios', label: 'iPhone 17' }, { id: 'duo-closed', platform: 'ios', label: 'iPhone Duo (closed)' },
{ id: 'duo-half-open', platform: 'ios', label: 'iPhone Duo (half-open)' }, { id: 'duo-open', platform: 'ios', label: 'iPhone Duo (open)' },
{ id: 'android', platform: 'android', label: 'Android' }];
const footnote = 'Added text “Changes apply live while playing. Base resolution sets the step grid for lanes you add next.”.';
const footnoteText = 'Added text “Changes apply live while playing\\. Base resolution sets the step grid for lanes you add next\\.”';
function euxyReview(target, options = {}) {
const own = { sourceImpact: ['own'], sourceScope: 'screen' };
const matched = { verdict: 'unchanged', reason: 'matched', mismatchPercentage: 0, threshold: 0.1, tool: '[email protected]' };
const changed = (description, mismatchPercentage) => ({ ...own,
review: { status: 'modified', description, facts: [{ category: 'element', templateId: 'element.added', bullet: description }] },
visual: { verdict: 'modified', reason: 'repeated-cross-build-delta', mismatchPercentage, threshold: 0.1, tool: '[email protected]' } });
const inTreeOnly = { ...changed(footnote, 0), visual: matched };
if (options.offScreen !== false) inTreeOnly.review.offScreen = true;
if (options.diffFailed) inTreeOnly.visual = { ...matched, reason: 'diff-failed' };
const unseen = { ...own, review: { status: 'unchanged', description: 'No visible change' }, visual: matched };
const tempo = target === 'duo-closed' ? inTreeOnly : ['duo-half-open', 'duo-open'].includes(target) ? unseen : changed(footnote, 0.11);
const rename = changed('Added text “Pattern name”.', 0.16);
const schemes = target === 'android' ? [''] : ['', '@dark'];
return review(schemes.flatMap(scheme => [
surface(`tempo${scheme}`, 'modified', { route: '/tempo', extra: structuredClone(tempo) }),
surface(`rename-pattern${scheme}`, 'modified', { route: '/rename-pattern', extra: structuredClone(rename) }),
]));
}
async function euxyRows(reviewOf = euxyReview) {
const { platformSectionMerge } = await import('../../dist/report/pr-comment-body.js');
let body;
for (const target of euxyTargets) body = platformSectionMerge(state({ artifactOnly: true, target, targets: euxyTargets, review: reviewOf(target.id),
build: { ...state().build, platform: target.platform }, runtime: { ...state().runtime, platform: target.platform, deviceModel: target.label } }), target.id).merge(body);
return { body, rows: listOf(body).map(line => line.replace(/\(<[^>]+>\)/, '')) };
}
test('a change whose elements run past the screen on a device is named off-screen there, beside the devices that show it (euxy#172)', async () => {
const { body, rows } = await euxyRows();
assert.deepEqual(rows, [
`- [\`/tempo\`] · ${footnoteText} · iPhone 17, Android only · off-screen on iPhone Duo \\(closed\\)`,
`- [\`/rename-pattern\`] · Added text “Pattern name”`,
`- [\`/tempo\`] (dark) · ${footnoteText} · iPhone 17 only · off-screen on iPhone Duo \\(closed\\)`,
`- [\`/rename-pattern\`] (dark) · Added text “Pattern name” · not on Android`,
], 'the half-open and opened Duo have no trace of it, so they stay among the devices it is not on');
const { readCommentRecords } = await import('../../dist/report/pr-comment-body.js');
assert.deepEqual(readCommentRecords(body).current['duo-closed'].changes.filter(change => change.offScreen).map(change => change.id), ['tempo', 'tempo@dark'],
'the mark survives the comment round trip');
});
test('without geometry that puts it past the screen, or with a failed comparison, a device keeps today\'s wording', async () => {
// The review euxy#172 published: the closed Duo's change is accessibility-only, with nothing saying where its elements are.
const { rows } = await euxyRows(target => euxyReview(target, { offScreen: false }));
assert.equal(rows[0], `- [\`/tempo\`] · ${footnoteText} · not on iPhone Duo \\(half\\-open\\), iPhone Duo \\(open\\)`);
assert.equal(rows[2], `- [\`/tempo\`] (dark) · ${footnoteText} · iPhone 17, iPhone Duo \\(closed\\) only`);
// A comparison that failed is no verdict at all: the closed Duo is not counted among the devices with the change.
const failed = await euxyRows(target => euxyReview(target, { diffFailed: true }));
assert.match(failed.rows[0], / · on iPhone 17, Android$/);
assert.doesNotMatch(failed.rows.join('\n'), /off-screen/);
});
test('a change one platform has and another lacks keeps "only" and "not on"', async () => {
// An iOS-only branch in a shared screen file: Android's capture of the screen shows no change, and says nothing off-screen.
const iosOnly = target => {
const reviewed = euxyReview(target);
if (target === 'android') for (const entry of reviewed.surfaces) Object.assign(entry, { review: { status: 'unchanged', description: 'No visible change' },
visual: { verdict: 'unchanged', reason: 'matched', mismatchPercentage: 0, threshold: 0.1, tool: '[email protected]' } });
return reviewed;
};
const { rows } = await euxyRows(iosOnly);
assert.equal(rows[0], `- [\`/tempo\`] · ${footnoteText} · iPhone 17 only · off-screen on iPhone Duo \\(closed\\)`);
assert.match(rows.find(row => row.startsWith('- [`/rename-pattern`] ·')), / · not on Android$/);
});
test('several targets share a comment: named in config order in the subtitle, each linking its report, keyed by id in the records and history', async () => {
const { platformSectionMerge, readCommentRecords } = await import('../../dist/report/pr-comment-body.js');
const [iphone, closed, open, android] = duoTargets;
@@ -320,3 +320,15 @@ test("an excluded surface carries no capture evidence", () => {
excluded.surfaces[1].excluded = false;
assert.throws(() => assertPublicReview(excluded), /excluded surface carries capture evidence/);
});
test("a review says its change is off-screen only on a modified verdict whose screenshots match", () => {
const offScreen = (status, verdict) => {
const input = review();
input.surfaces[0].review = { status, description: status === "modified" ? "Added text “lanes you add next”." : "No visible change", offScreen: true };
input.surfaces[0].visual = { verdict, reason: verdict === "unchanged" ? "matched" : "repeated-cross-build-delta", mismatchPercentage: verdict === "unchanged" ? 0 : 3, threshold: 0.1, tool: "[email protected]" };
return input;
};
assert.doesNotThrow(() => assertPublicReview(offScreen("modified", "unchanged")));
assert.throws(() => assertPublicReview(offScreen("modified", "modified")), /inconsistent/);
assert.throws(() => assertPublicReview(offScreen("unchanged", "unchanged")), /inconsistent/);
});
+13
View File
@@ -97,3 +97,16 @@ test("accessibility fingerprints ignore Android system UI nodes", async () => {
assert.equal(sameApp.accessibility, base.accessibility);
assert.notEqual(changedApp.accessibility, base.accessibility);
});
test("accessibility fingerprints are versioned, and leave out the system keyboard and which field has the focus", async () => {
const snapshot = (focused, keyboard) => JSON.stringify({ success: true, data: { nodes: [
{ index: 0, type: "Application", label: "euxy", rect: { x: 0, y: 0, width: 402, height: 874 } },
{ index: 1, parentIndex: 0, type: "TextField", label: "Pattern name", ...(focused ? { focused: true } : {}), rect: { x: 23, y: 386, width: 355, height: 50 } },
...(keyboard ? [{ index: 2, parentIndex: 0, type: "Keyboard", rect: { x: 0, y: 583, width: 402, height: 233 } },
{ index: 3, parentIndex: 2, type: "Key", label: "Q", rect: { x: 5, y: 590, width: 39, height: 54 } }] : []),
] } });
const resting = await fingerprintState({ snapshotText: snapshot(false, false) });
assert.match(resting.accessibility, /^a2:[0-9a-f]{64}$/);
assert.equal((await fingerprintState({ snapshotText: snapshot(true, false) })).accessibility, resting.accessibility, "focus alone does not change the tree");
assert.equal((await fingerprintState({ snapshotText: snapshot(true, true) })).accessibility, resting.accessibility, "nor does the keyboard");
});