Compare commits

..
20 changed files with 72 additions and 967 deletions

No files matched your search

-35
View File
@@ -198,41 +198,6 @@ npx @expo/explore explore:session <session-id> --env-file path/.env.eas-simulato
</details>
<details>
<summary><strong>Take the same steps on every run and device</strong></summary>
A model picks its own path, so two runs (or an iPhone and an Android phone) can reach different screens. To get the same screens every time, write the steps down once, for example after a model run showed you the app, and pass them with `--replay`. No model login or API key is needed.
```sh
npx @expo/explore explore:binary ./my-app.app --replay steps.json
npx @expo/explore explore:session --replay steps.json # an iPhone Duo pose, or any session you set up by hand
```
```json
{
"steps": [
{ "capture": "Home" },
{ "action": "tap", "label": "Settings" },
{ "capture": "Settings" },
{ "action": "tap", "label": "Dark" },
{ "capture": "Settings", "state": "dark" },
{ "action": "tap", "label": "Learn", "platforms": ["ios"] },
{ "capture": "Learn", "platforms": ["ios"] },
{ "action": "tap", "label": "Got it", "optional": true }
]
}
```
- An action is one of `tap` or `fill` (with `label` or `identifier`), `type`, `scroll` (`direction`), `wait` (`milliseconds`), `open` (`url`), `back`, and `dismiss-keyboard`, as in the model's own tools. The same safety rules apply: destructive or app-leaving taps are refused. A step takes only the fields listed here; any other field (a typo such as `platform`) is an error.
- `{ "capture": "Title" }` photographs the current screen under that title; add `"state"` for another state of a screen already captured. A capture of a screen the plan already captured under another title is reported as looking the same, and the screen keeps its first title.
- `"platforms": ["ios"]` limits a step to those platforms, for screens one platform does not have. `"optional": true` skips an action that fails (a one-time dialog) instead of stopping; the summary lists the steps it skipped.
- A `fill` or `type` text can be a `{{secret:NAME}}`, as in `--hints`: Explore types the value of the environment variable `NAME`, and a variable that is not set fails before a device starts.
- If a required action fails, the run stops there and the report still shows what was captured, with a gap naming the step and the screens it did not reach.
- The screen and action budgets still apply. A plan with more captures or actions than they allow is refused before a device starts, as is one with no steps for the platform; raise `--max-screens` and `--max-actions` for a long plan. If a budget still runs out (each screen the plan reaches also counts), the gap names the planned screens not reached.
- `--provider`, `--model` and `--hints` only steer a model, so they cannot be combined with `--replay`.
</details>
<details>
<summary><strong>What a run without source shows</strong></summary>
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@expo/explore",
"version": "0.20.3",
"version": "0.20.2",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@expo/explore",
"version": "0.20.3",
"version": "0.20.2",
"license": "MIT",
"dependencies": {
"@expo/repack-app": "0.10.1",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@expo/explore",
"version": "0.20.3",
"version": "0.20.2",
"description": "Source-driven mobile app maps, deterministic CI capture, and visual reviews",
"license": "MIT",
"type": "module",
-9
View File
@@ -202,15 +202,6 @@
"mimeType": { "type": "string", "minLength": 1 },
"width": { "type": "integer", "minimum": 0 },
"height": { "type": "integer", "minimum": 0 },
"turned": {
"type": "object",
"additionalProperties": false,
"required": ["width", "height"],
"properties": {
"width": { "type": "number", "exclusiveMinimum": 0 },
"height": { "type": "number", "exclusiveMinimum": 0 }
}
},
"createdAt": { "$ref": "#/$defs/timestamp" }
}
},
-4
View File
@@ -33,10 +33,6 @@ Platform, bundle id, app name and version are read from the archive; do not pass
Budgets default to 15 minutes, 40 screens, 150 actions (`--max-duration-minutes`, `--max-screens`, `--max-actions`). Steer the model with `--hints "focus on the onboarding and settings"`, skip onboarding with `--launch-url myapp://home`, and allow a specific risky control with `--allow-destructive "Clear cache"`.
## The same steps on every run: `--replay`
To compare runs, or an iPhone with an Android phone, screen for screen, pass `--replay steps.json` to `explore:binary` or `explore:session`: Explore takes the steps in the file instead of a model's, and no model access is needed. The format is in the README ("Take the same steps on every run and device"): `{ "steps": [{ "capture": "Home" }, { "action": "tap", "label": "Settings" }, { "capture": "Settings" }] }`, with `platforms` to limit a step and `optional` for an action that may fail. The plan is checked before a device starts: unknown fields, a plan with no steps for the platform, more captures or actions than `--max-screens` / `--max-actions`, and a `{{secret:NAME}}` in a `fill` or `type` whose variable is not set are all refused (`REPLAY_PLAN_INVALID`, `REPLAY_PLAN_EMPTY`, `REPLAY_PLAN_OVER_BUDGET`, `EXPLORATION_SECRET_MISSING`). `--provider`, `--model` and `--hints` cannot be combined with it (`REPLAY_OPTIONS_IGNORED`).
## Sign-in and other app state
Without help, the model treats a sign-in wall as the end of the road: it captures it and stops (`blocked`). Before running, read what you can about the app (its source if you have it, its first screens otherwise) and decide what it needs.
+27 -59
View File
@@ -7,7 +7,6 @@ import type { CaptureSetupAction } from "../config/explore-config.js";
import { createExplorationInputs, explorationSecretNames } from "../config/exploration-inputs.js";
import type { ExploreStore, PreparedArtifact } from "../map/store.js";
import { CiError } from "../shared/ci-errors.js";
import { turnedClockwise } from "../shared/png-evidence.js";
import type { ActionKind, ActionRisk, Artifact, Controller, ElementRole, ExploreMap, ObservedElement, Observation, Screen } from "../types.js";
/** One look at the device: accessibility always, a PNG when the host could take a safe one. */
@@ -22,15 +21,13 @@ export type BlackBoxFrame = {
imageUnavailable?: { code: string; message: string };
/** The foreground app differed from the explored one; the host already brought it back. */
leftApp?: boolean;
/** The image is on a portrait panel's axes under a landscape UI (an opened iPhone Duo), as the tree is; the report stores it turned upright. */
landscape?: boolean;
};
export type BlackBoxAction = CaptureSetupAction | { action: "back" } | { action: "dismiss-keyboard" } | { action: "type"; text: string }
| { action: "tap"; frameId: string; x: number; y: number };
export type BlackBoxExplorerOptions = {
session: Pick<CiAgentSession, "run">;
session: CiAgentSession;
store: ExploreStore;
map: ExploreMap;
app: { name: string; bundleId: string; platform: "ios" | "android"; version?: string };
@@ -52,8 +49,6 @@ export type BlackBoxExplorerOptions = {
appIcon?: Buffer;
/** App-test inputs the operator hints reference as {{secret:NAME}}: the model and the map only ever hold the reference. */
inputs?: ReturnType<typeof createExplorationInputs>;
/** A sentence a budget stop's gap adds to its own, read once the session has stopped: a replay names the planned screens it did not reach. */
budgetNote?: () => string | undefined;
};
export type BlackBoxExplorerResult = {
@@ -72,12 +67,6 @@ const EXTERNAL = /\b(open in|safari|chrome|browser|app store|play store|share|ma
const VOLATILE = /\d|[:%$€£]|\b(am|pm)\b/gi;
const INTERACTIVE = new Set(["button", "link", "cell", "switch", "tab", "menuitem", "textfield", "securetextfield", "searchfield", "textview", "edittext", "slider", "checkbox", "radiobutton", "segmentedcontrol", "toggle"]);
/** What the tools accept: their schemas, the checks on each call, and a replay plan checked before any device starts share these. */
export const TOOL_LIMITS = { selector: 512, text: 2000, url: 2000, waitMs: 15_000, title: 80, state: 80, description: 500, summary: 4000, reason: 1000 } as const;
/** A screen seen after an action but never named keeps this title until the run ends and names it after the path to it. */
export const isPlaceholderTitle = (title: string): boolean => /^Screen \d+$/.test(title);
const selectorProperties = {
label: { type: "string", description: "Exact visible label of the target node, copied from the latest observation." },
identifier: { type: "string", description: "Exact accessibility identifier of the target node, when it has one." },
@@ -92,22 +81,22 @@ const tools: CiAgentTool[] = [
inputSchema: { type: "object", properties: {
action: { type: "string", enum: ["tap", "fill", "type", "scroll", "wait", "back", "dismiss-keyboard", "open"] },
...selectorProperties,
text: { type: "string", maxLength: TOOL_LIMITS.text }, url: { type: "string", maxLength: TOOL_LIMITS.url },
text: { type: "string", maxLength: 2000 }, url: { type: "string", maxLength: 2000 },
direction: { type: "string", enum: ["up", "down", "left", "right", "top", "bottom"] },
milliseconds: { type: "integer", minimum: 1, maximum: TOOL_LIMITS.waitMs },
milliseconds: { type: "integer", minimum: 1, maximum: 15_000 },
frameId: { type: "string", description: "For a coordinate tap only: the latest observed frame." },
x: { type: "number", minimum: 0, exclusiveMaximum: 1 }, y: { type: "number", minimum: 0, exclusiveMaximum: 1 },
target: { type: "string", minLength: 1, maxLength: 80, description: "For a coordinate tap: what is being tapped, as a user would name it (\"search icon in the header\", \"first manga cover\")." },
}, required: ["action"], additionalProperties: false } },
{ name: "capture", description: "Name the latest observed frame (observe with includeImage) as a screen of the map, with a short human title. A screen is photographed the moment it first appears; capture gives it its title, and with state records another notable state of it.",
inputSchema: { type: "object", properties: {
frameId: { type: "string" }, title: { type: "string", minLength: 1, maxLength: TOOL_LIMITS.title },
description: { type: "string", maxLength: TOOL_LIMITS.description }, state: { type: "string", maxLength: TOOL_LIMITS.state, description: "A variant of an already captured screen, such as 'empty' or 'menu open'." },
frameId: { type: "string" }, title: { type: "string", minLength: 1, maxLength: 80 },
description: { type: "string", maxLength: 500 }, state: { type: "string", maxLength: 80, description: "A variant of an already captured screen, such as 'empty' or 'menu open'." },
}, required: ["frameId", "title"], additionalProperties: false } },
{ name: "finish", description: "End the exploration when every reachable screen has been captured or the budget is nearly used. Include a summary of what the app does and what was not reached.",
inputSchema: { type: "object", properties: { summary: { type: "string", minLength: 1, maxLength: TOOL_LIMITS.summary } }, required: ["summary"], additionalProperties: false } },
inputSchema: { type: "object", properties: { summary: { type: "string", minLength: 1, maxLength: 4000 } }, required: ["summary"], additionalProperties: false } },
{ name: "stop_blocked", description: "Stop because the app cannot be explored further (login wall without credentials, crash loop, permission prompt that cannot be dismissed). Keeps the evidence so far.",
inputSchema: { type: "object", properties: { reason: { type: "string", minLength: 1, maxLength: TOOL_LIMITS.reason } }, required: ["reason"], additionalProperties: false } },
inputSchema: { type: "object", properties: { reason: { type: "string", minLength: 1, maxLength: 1000 } }, required: ["reason"], additionalProperties: false } },
];
const INSTRUCTIONS = `You are exploring a prebuilt mobile app on a remote simulator through accessibility and screenshots. Nobody gave you its source. Your job is to produce a map of its screens: visit every reachable screen, capture each one once with a clear title, and try each distinct kind of control so transitions are recorded.
@@ -423,7 +412,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
}
/** Seen after an action but never named by the model (its screenshot was taken on arrival). */
const isPlaceholder = (screen: Screen): boolean => isPlaceholderTitle(screen.title);
const isPlaceholder = (screen: Screen): boolean => /^Screen \d+$/.test(screen.title);
/** Fold an uncaptured placeholder into the screen the model says it is. */
const mergeScreens = (from: Screen, into: Screen): void => {
for (const item of map.actionAttempts) {
@@ -448,22 +437,8 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
if (currentScreenId === from.id) currentScreenId = into.id;
};
/**
* The bytes a frame's screenshot is stored as, and their digest: an opened Duo's frame is turned upright once, however
* often the frame is looked up or recorded. Keyed by the image, which every copy of the frame shares.
*/
const storedImages = new WeakMap<Buffer, { png: Buffer; sha256: string }>();
const storedImage = (image: Buffer, landscape: boolean): { png: Buffer; sha256: string } => {
let stored = storedImages.get(image);
if (!stored) {
const png = landscape ? turnedClockwise(image) : image;
stored = { png, sha256: createHash("sha256").update(png).digest("hex") };
storedImages.set(image, stored);
}
return stored;
};
const screenWithImage = (sha: string, except: Screen): Screen | undefined => {
const screenWithImage = (png: Buffer, except: Screen): Screen | undefined => {
const sha = createHash("sha256").update(png).digest("hex");
const artifact = map.artifacts.find(item => item.kind === "screenshot" && item.sha256 === sha);
if (!artifact) return undefined;
const controls = structural(evidence.get(except.id)?.parts ?? new Set());
@@ -516,7 +491,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
screen = { id: `scr_${key.slice(0, 16)}`, title: title ?? systemDialogTitle(frame.nodes) ?? `Screen ${map.screens.length + 1}`, start: map.screens.length === 0, terminal: false,
depth, firstSeenAt: frame.capturedAt, lastSeenAt: frame.capturedAt, visits: 0, variantIds: [], observationIds: [], elementIds: [] };
map.screens.push(screen); screensByKey.set(key, screen);
} else if (title && isPlaceholderTitle(screen.title)) screen.title = title;
} else if (title && /^Screen \d+$/.test(screen.title)) screen.title = title;
screen.visits += 1; screen.lastSeenAt = frame.capturedAt;
const seen = evidence.get(screen.id) ?? { parts: new Set<string>(), loadingOnly: true };
for (const part of loose) seen.parts.add(part);
@@ -554,13 +529,11 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
let screenshotArtifactId: string | undefined;
if (withImage && frame.png) {
// Hashed here rather than trusting the daemon's digest: the report validates artifacts against their bytes.
const { png, sha256: sha } = storedImage(frame.png, frame.landscape === true);
const extent = frame.landscape ? frameExtent(frame.nodes) : undefined;
const png = frame.png, sha = createHash("sha256").update(png).digest("hex");
let artifact = map.artifacts.find(item => item.kind === "screenshot" && item.sha256 === sha);
if (!artifact) {
artifact = { id: `art_${randomUUID().slice(0, 12)}`, kind: "screenshot" as const, path: `artifacts/screenshots/${sha}.png`, sha256: sha,
bytes: png.length, mimeType: "image/png", ...(frame.width && frame.height ? frame.landscape ? { width: frame.height, height: frame.width } : { width: frame.width, height: frame.height } : {}),
...(extent ? { turned: extent } : {}), createdAt: frame.capturedAt };
bytes: png.length, mimeType: "image/png", ...(frame.width ? { width: frame.width } : {}), ...(frame.height ? { height: frame.height } : {}), createdAt: frame.capturedAt };
map.artifacts.push(artifact); prepared.push({ artifact, bytes: png, destination: join(store.root, artifact.path) });
}
screenshotArtifactId = artifact.id;
@@ -626,7 +599,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
// The same pixels as a screen already seen, with the same controls, is that screen: a tree read while Android was
// still exposing a text node (AudioScape's playlist title missing from one read of two identical frames) is not new.
if (created && latest.png) {
const twin = screenWithImage(storedImage(latest.png, latest.landscape === true).sha256, screen);
const twin = screenWithImage(latest.png, screen);
if (twin) { mergeScreens(screen, twin); screen = twin; created = false; }
}
// Every screen gets an observation the moment it is seen, so elements and action attempts always point at a
@@ -652,7 +625,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
const actionFor = (input: Record<string, unknown>): { action: BlackBoxAction; kind: ActionKind; label: string; risk: ActionRisk } => {
const action = String(input.action);
const selector = input.identifier !== undefined ? { identifier: text(input.identifier, "identifier", TOOL_LIMITS.selector) } : input.label !== undefined ? { label: text(input.label, "label", TOOL_LIMITS.selector) } : undefined;
const selector = input.identifier !== undefined ? { identifier: text(input.identifier, "identifier", 512) } : input.label !== undefined ? { label: text(input.label, "label", 512) } : undefined;
const label = selector ? ("identifier" in selector ? selector.identifier : selector.label) : action;
const risk = selector ? riskOf(label, allow) : "safe";
switch (action) {
@@ -664,13 +637,13 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
}
if (!selector) throw failure("AGENT_ACTION_INVALID", "tap needs a label, an identifier, or frame coordinates.");
return { action: { action: "tap", ...selector } as BlackBoxAction, kind: "press", label, risk };
case "fill": if (!selector) throw failure("AGENT_ACTION_INVALID", "fill needs a label or identifier."); return { action: { action: "fill", text: text(input.text, "text", TOOL_LIMITS.text), ...selector } as BlackBoxAction, kind: "fill", label, risk };
case "type": return { action: { action: "type", text: text(input.text, "text", TOOL_LIMITS.text) }, kind: "fill", label: "type", risk: "safe" };
case "fill": if (!selector) throw failure("AGENT_ACTION_INVALID", "fill needs a label or identifier."); return { action: { action: "fill", text: text(input.text, "text", 2000), ...selector } as BlackBoxAction, kind: "fill", label, risk };
case "type": return { action: { action: "type", text: text(input.text, "text", 2000) }, kind: "fill", label: "type", risk: "safe" };
case "scroll": return { action: { action: "scroll", direction: input.direction as "up" }, kind: "swipe", label: `scroll ${String(input.direction)}`, risk: "safe" };
case "wait": return { action: { action: "wait", milliseconds: Number(input.milliseconds) || 1000 }, kind: "other", label: "wait", risk: "safe" };
case "back": return { action: { action: "back" }, kind: "back", label: "back", risk: "safe" };
case "dismiss-keyboard": return { action: { action: "dismiss-keyboard" }, kind: "dismiss", label: "dismiss keyboard", risk: "safe" };
case "open": return { action: { action: "open", url: text(input.url, "url", TOOL_LIMITS.url) }, kind: "deep-link", label: text(input.url, "url", TOOL_LIMITS.url), risk: "unknown" };
case "open": return { action: { action: "open", url: text(input.url, "url", 2000) }, kind: "deep-link", label: text(input.url, "url", 2000), risk: "unknown" };
default: throw failure("AGENT_ACTION_INVALID", `Unknown action ${action}.`);
}
};
@@ -734,12 +707,8 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
const role = roleOf(under);
const label = str(under.label) ?? "";
const what = role === "field" ? "field" : role === "button" || role === "tab" || role === "link" ? (cleanLabel(label) ? role : `icon ${role}`) : /\p{L}/u.test(cleanLabel(label)) ? "item" : "icon";
// The place is read off the screen's stored screenshot, which an opened Duo's frames are turned upright for: there the
// portrait point (x, y) shows at (1 - y, x). A frame read without an image says nothing, so the screen's screenshot is asked.
const turned = latest!.landscape || map.observations.some(item => item.screenId === currentScreenId && map.artifacts.find(artifact => artifact.id === item.screenshotArtifactId)?.turned);
const shown = turned ? { x: 1 - point.y, y: point.x } : point;
const row = shown.y < 1 / 3 ? "top" : shown.y < 2 / 3 ? "" : "bottom";
const column = shown.x < 1 / 3 ? "left" : shown.x < 2 / 3 ? "" : "right";
const row = point.y < 1 / 3 ? "top" : point.y < 2 / 3 ? "" : "bottom";
const column = point.x < 1 / 3 ? "left" : point.x < 2 / 3 ? "" : "right";
const where = [row, column].filter(Boolean).join(" ") || "centre";
return `${what}, ${where}`;
};
@@ -780,10 +749,10 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
const frameId = text(input.frameId, "frameId", 256);
if (!latest?.png && latest?.imageUnavailable?.code === "CAPTURE_SENSITIVE_INPUT") throw failure("CAPTURE_SENSITIVE_INPUT", latest.imageUnavailable.message, true);
if (!latest?.png || latest.frameId !== frameId) throw failure("AGENT_FRAME_STALE", "Observe with an image first; capture must name the latest frame.", true);
const title = text(input.title, "title", TOOL_LIMITS.title).trim();
const title = text(input.title, "title", 80).trim();
if (captures >= options.maxScreens) { stop("budget-screens"); abort.abort(failure("EXPLORATION_BUDGET_EXHAUSTED", "Screen budget exhausted.")); throw abort.signal.reason; }
const { screen } = locate(latest, title);
const state = str(input.state, TOOL_LIMITS.state);
const state = str(input.state, 80);
if (!state && map.observations.some(item => item.screenId === screen.id && item.screenshotArtifactId)) {
if (title) screen.title = title;
return toolResult({ ok: true, duplicate: true, screenId: screen.id, title: screen.title, message: "This screen is already captured (its title is updated); pass state to record a variant." });
@@ -863,9 +832,9 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
await persist();
return toolResult({ ok: !error, outcome, ...(error ? { error } : {}), ...view, ...progress().budget });
}
if (name === "finish") { stop("finished", text(input.summary, "summary", TOOL_LIMITS.summary)); abort.abort(new Error("finished")); return toolResult({ ok: true }); }
if (name === "finish") { stop("finished", text(input.summary, "summary", 4000)); abort.abort(new Error("finished")); return toolResult({ ok: true }); }
if (name === "stop_blocked") {
const reason = text(input.reason, "reason", TOOL_LIMITS.reason);
const reason = text(input.reason, "reason", 1000);
map.gaps.push({ id: `gap_${randomUUID().slice(0, 8)}`, kind: "coverage", status: "open", message: reason, ...(currentScreenId ? { screenId: currentScreenId } : {}), createdAt: timestamp() });
await persist();
stop("blocked", reason); abort.abort(new Error("blocked")); return toolResult({ ok: true });
@@ -887,9 +856,8 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
if (!result) stop(budget ? "budget-time" : "model-error", undefined, budget ? undefined : fatal ?? error);
}
if (result!.stopReason !== "finished" && result!.stopReason !== "blocked") {
const note = result!.stopReason === "model-error" ? undefined : options.budgetNote?.();
map.gaps.push({ id: `gap_${randomUUID().slice(0, 8)}`, kind: result!.stopReason === "model-error" ? "other" : "coverage", status: "open",
message: result!.stopReason === "model-error" ? `The exploration stopped on an error: ${result!.error instanceof Error ? result!.error.message.slice(0, 300) : "unknown"}` : `The exploration stopped at its ${result!.stopReason.replace("budget-", "")} budget with ${map.frontier.filter(item => item.status === "pending").length} controls unexplored.${note ? ` ${note}` : ""}`,
message: result!.stopReason === "model-error" ? `The exploration stopped on an error: ${result!.error instanceof Error ? result!.error.message.slice(0, 300) : "unknown"}` : `The exploration stopped at its ${result!.stopReason.replace("budget-", "")} budget with ${map.frontier.filter(item => item.status === "pending").length} controls unexplored.`,
createdAt: timestamp() });
}
for (const item of map.frontier) if (item.status === "claimed") item.status = "pending";
@@ -902,10 +870,10 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
function nameUncapturedScreens(map: ExploreMap): void {
const titles = new Map(map.screens.map(screen => [screen.id, screen.title]));
for (const screen of map.screens) {
if (!isPlaceholderTitle(screen.title)) continue;
if (!/^Screen \d+$/.test(screen.title)) continue;
const arrival = map.actionAttempts.find(attempt => attempt.targetScreenId === screen.id && attempt.outcome === "transition" && attempt.sourceScreenId !== screen.id);
const from = arrival ? titles.get(arrival.sourceScreenId) : undefined;
if (!arrival || !from || isPlaceholderTitle(from)) continue;
if (!arrival || !from || /^Screen \d+$/.test(from)) continue;
const step = arrival.action.input ? "typing" : arrival.action.label;
screen.title = `${from} → ${step}`.slice(0, 80);
titles.set(screen.id, screen.title);
-300
View File
@@ -1,300 +0,0 @@
import { readFile } from "node:fs/promises";
import { CiError } from "../shared/ci-errors.js";
import type { CiAgentSessionRunRequest, CiAgentSessionRunResult } from "../capture/agent/session.js";
import { SETUP_SELECTOR_MAX, isSetupInput, isSetupSelector } from "../capture/setup-actions.js";
import type { CaptureSetupAction } from "../config/explore-config.js";
import { explorationSecretNames } from "../config/exploration-inputs.js";
import { TOOL_LIMITS, isPlaceholderTitle } from "./explorer.js";
export type ReplayPlatform = "ios" | "android";
type Selector = { label: string; identifier?: never } | { identifier: string; label?: never };
type ReplayAction =
| ({ action: "tap" } & Selector)
| ({ action: "fill"; text: string } & Selector)
| { action: "type"; text: string }
| { action: "scroll"; direction: "up" | "down" | "left" | "right" | "top" | "bottom" }
| { action: "wait"; milliseconds: number }
| { action: "open"; url: string }
| { action: "back" }
| { action: "dismiss-keyboard" };
type ReplayCapture = { capture: string; state?: string; description?: string };
/** `platforms` limits a step to those platforms; an `optional` action that fails is skipped instead of ending the replay. */
export type ReplayStep = (ReplayCapture & { platforms?: ReplayPlatform[] }) | (ReplayAction & { platforms?: ReplayPlatform[]; optional?: boolean });
export type ReplayPlan = { steps: ReplayStep[] };
export type ReplaySession = {
readonly provider: "replay";
readonly model: string;
preflight(): Promise<void>;
run(request: CiAgentSessionRunRequest): Promise<CiAgentSessionRunResult>;
/** After a budget stop, the planned screens the run did not reach, for the explorer's gap. */
budgetNote(): string | undefined;
};
const DIRECTIONS = new Set(["up", "down", "left", "right", "top", "bottom"]);
/**
* The fields each kind of step takes. Anything else is refused: a typo (`platform`, `optinal`) would otherwise do nothing,
* and a field `act` reads another way (a coordinate tap's `frameId`, `x`, `y` and `target`) would change what is tapped.
*/
const FIELDS = new Map<string, readonly string[]>([
["capture", ["capture", "state", "description", "platforms"]],
["tap", ["action", "label", "identifier", "platforms", "optional"]],
["fill", ["action", "label", "identifier", "text", "platforms", "optional"]],
["type", ["action", "text", "platforms", "optional"]],
["scroll", ["action", "direction", "platforms", "optional"]],
["wait", ["action", "milliseconds", "platforms", "optional"]],
["open", ["action", "url", "platforms", "optional"]],
["back", ["action", "platforms", "optional"]],
["dismiss-keyboard", ["action", "platforms", "optional"]],
]);
const ACTIONS = [...FIELDS.keys()].filter(kind => kind !== "capture");
function invalid(message: string): CiError {
return new CiError({ code: "REPLAY_PLAN_INVALID", stage: "exploration", message, retryable: false,
nextAction: "Fix the replay plan: a JSON object with a steps array of actions ({\"action\":\"tap\",\"label\":\"Settings\"}) and captures ({\"capture\":\"Settings\"})." });
}
const nonEmpty = (value: unknown, max: number): value is string => typeof value === "string" && value.trim().length > 0 && value.length <= max;
function parseStep(step: unknown, at: string): ReplayStep {
if (!step || typeof step !== "object" || Array.isArray(step)) throw invalid(`${at} is not an object.`);
const entry = step as Record<string, unknown>;
const kind = "capture" in entry ? "capture" : typeof entry.action === "string" && FIELDS.has(entry.action) ? entry.action : undefined;
if (!kind) throw invalid(`${at} needs a capture title or an action (${ACTIONS.join(", ")}).`);
const fields = FIELDS.get(kind)!;
const unknown = Object.keys(entry).find(key => !fields.includes(key));
if (unknown !== undefined) throw invalid(`${at} (${kind}) has a field ${JSON.stringify(unknown.slice(0, 40))} it does not take; a ${kind} step takes ${fields.join(", ")}.`);
if (entry.platforms !== undefined && (!Array.isArray(entry.platforms) || !entry.platforms.length || entry.platforms.some(item => item !== "ios" && item !== "android"))) {
throw invalid(`${at}: platforms must list ios and/or android.`);
}
if (entry.optional !== undefined && typeof entry.optional !== "boolean") throw invalid(`${at}: optional must be true or false.`);
const extra = { ...(entry.platforms ? { platforms: [...entry.platforms] as ReplayPlatform[] } : {}), ...(entry.optional !== undefined ? { optional: entry.optional as boolean } : {}) };
if (kind === "capture") {
if (!nonEmpty(entry.capture, TOOL_LIMITS.title)) throw invalid(`${at}: capture must be a screen title of at most ${TOOL_LIMITS.title} characters.`);
if (entry.state !== undefined && !nonEmpty(entry.state, TOOL_LIMITS.state)) throw invalid(`${at}: state must be a string of at most ${TOOL_LIMITS.state} characters.`);
if (entry.description !== undefined && !nonEmpty(entry.description, TOOL_LIMITS.description)) throw invalid(`${at}: description must be a string of at most ${TOOL_LIMITS.description} characters.`);
return { capture: entry.capture.trim(), ...(entry.state !== undefined ? { state: entry.state as string } : {}),
...(entry.description !== undefined ? { description: entry.description as string } : {}), ...extra };
}
// The device selects by the setup executor's rules, which are stricter than what `act` itself accepts.
const selector = (): Selector => {
const keys = (["label", "identifier"] as const).filter(key => entry[key] !== undefined);
if (keys.length !== 1) throw invalid(`${at}: ${kind} needs exactly one of label or identifier.`);
const key = keys[0]!, value = entry[key];
if (!isSetupSelector(value) || value.length > TOOL_LIMITS.selector) throw invalid(`${at}: ${key} must be 1–${SETUP_SELECTOR_MAX} characters without control characters.`);
return key === "label" ? { label: value } : { identifier: value };
};
const input = (): string => {
if (!nonEmpty(entry.text, TOOL_LIMITS.text) || !isSetupInput(entry.text)) throw invalid(`${at}: ${kind} needs text: one line of at most ${TOOL_LIMITS.text} characters.`);
try { explorationSecretNames(entry.text); }
catch (error) { throw invalid(`${at}: ${error instanceof Error ? error.message : String(error)} Write each input as {{secret:NAME}} with an uppercase environment variable name that is not a provider, Expo, or CI credential.`); }
return entry.text;
};
switch (kind) {
case "tap": return { action: "tap", ...selector(), ...extra };
case "fill": return { action: "fill", ...selector(), text: input(), ...extra };
case "type": return { action: "type", text: input(), ...extra };
case "scroll":
if (typeof entry.direction !== "string" || !DIRECTIONS.has(entry.direction)) throw invalid(`${at}: scroll needs a direction (${[...DIRECTIONS].join(", ")}).`);
return { action: "scroll", direction: entry.direction as "up", ...extra };
case "wait":
if (!(Number.isInteger(entry.milliseconds) && Number(entry.milliseconds) >= 1 && Number(entry.milliseconds) <= TOOL_LIMITS.waitMs)) {
throw invalid(`${at}: wait needs milliseconds between 1 and ${TOOL_LIMITS.waitMs}.`);
}
return { action: "wait", milliseconds: entry.milliseconds as number, ...extra };
case "open":
if (!nonEmpty(entry.url, TOOL_LIMITS.url)) throw invalid(`${at}: open needs a url of at most ${TOOL_LIMITS.url} characters.`);
return { action: "open", url: entry.url, ...extra };
default: return { action: kind as "back" | "dismiss-keyboard", ...extra };
}
}
/** Checks a plan and keeps only the fields each step takes, so what reaches `act` is what the plan documents. */
export function parseReplayPlan(value: unknown): ReplayPlan {
if (!value || typeof value !== "object" || !Array.isArray((value as { steps?: unknown }).steps)) throw invalid("The replay plan has no steps array.");
const unknown = Object.keys(value).find(key => key !== "steps");
if (unknown !== undefined) throw invalid(`The replay plan has a field ${JSON.stringify(unknown.slice(0, 40))} it does not take; it takes only steps.`);
const steps = (value as { steps: unknown[] }).steps;
if (!steps.length) throw invalid("The replay plan has no steps.");
return { steps: steps.map((step, index) => parseStep(step, `Step ${index + 1}`)) };
}
export async function readReplayPlan(path: string): Promise<ReplayPlan> {
let text: string;
try { text = await readFile(path, "utf8"); }
catch (error) { throw invalid(`The replay plan ${path} could not be read: ${error instanceof Error ? error.message : String(error)}`); }
let value: unknown;
try { value = JSON.parse(text); }
catch (error) { throw invalid(`The replay plan ${path} is not JSON: ${error instanceof Error ? error.message : String(error)}`); }
return parseReplayPlan(value);
}
/** The plan's fill and type steps, whose {{secret:NAME}} inputs must be set before any device starts, as the hints' are. */
export function replaySetup(plan: ReplayPlan): CaptureSetupAction[] {
return plan.steps.flatMap(step => "action" in step && (step.action === "fill" || step.action === "type") ? [step as CaptureSetupAction] : []);
}
const stepsFor = (plan: ReplayPlan, platform: ReplayPlatform) => plan.steps.map((step, index) => ({ step, number: index + 1 }))
.filter(({ step }) => !step.platforms || step.platforms.includes(platform));
/**
* Refuses, before any device starts, a plan with nothing to do on this platform or one the budgets cannot hold: a run that
* would stop at its budget partway through the plan is better raised or shortened up front.
*/
export function assertReplayFits(plan: ReplayPlan, platform: ReplayPlatform, limits: { maxScreens: number; maxActions: number; maxTurns: number }): void {
const steps = stepsFor(plan, platform);
if (!steps.length) {
throw new CiError({ code: "REPLAY_PLAN_EMPTY", stage: "exploration", message: `No step of the replay plan runs on ${platform}: each one is limited to ${platform === "ios" ? "android" : "ios"} by its platforms.`,
nextAction: `Add steps for ${platform} to the plan, or run it on the platform it was written for.` });
}
const captures = steps.filter(({ step }) => "capture" in step).length;
const actions = steps.filter(({ step }) => "action" in step && step.action !== "wait").length;
// A screen counts against the budget when it is first photographed, on arrival, and a capture is refused once the budget
// is used up, so the last capture needs one screen to spare.
const over = [
...(captures >= limits.maxScreens ? [{ text: `${captures} captures, which need a screen budget of at least ${captures + 1} (it is ${limits.maxScreens})`, flag: `--max-screens ${captures + 1}` }] : []),
...(actions > limits.maxActions ? [{ text: `${actions} actions, more than the action budget of ${limits.maxActions}`, flag: `--max-actions ${actions}` }] : []),
];
if (over.length) {
throw new CiError({ code: "REPLAY_PLAN_OVER_BUDGET", stage: "exploration", message: `On ${platform} the replay plan takes ${over.map(item => item.text).join(" and ")}.`,
nextAction: `Pass ${over.map(item => item.flag).join(" ")} (or more: each screen the plan reaches without capturing it also counts), or shorten the plan.` });
}
// Each capture is an observe and a capture call, each action one act call, and the run ends with get_progress and finish.
const turns = steps.length + captures + 2;
if (turns > limits.maxTurns) {
throw new CiError({ code: "REPLAY_PLAN_OVER_BUDGET", stage: "exploration", message: `On ${platform} the replay plan takes ${turns} tool calls, more than the ${limits.maxTurns} that the screen and action budgets allow.`,
nextAction: "Raise --max-actions, or use fewer wait steps." });
}
}
function describeAction(step: ReplayAction): string {
if ("label" in step && step.label) return `${step.action} "${step.label}"`;
if ("identifier" in step && step.identifier) return `${step.action} #${step.identifier}`;
if (step.action === "scroll") return `scroll ${step.direction}`;
return step.action;
}
/** Only the fields `act` documents for that action, whatever else the step object carries. */
function actInput(step: ReplayAction): Record<string, unknown> {
const selector = "label" in step && step.label !== undefined ? { label: step.label } : "identifier" in step && step.identifier !== undefined ? { identifier: step.identifier } : {};
switch (step.action) {
case "tap": return { action: "tap", ...selector };
case "fill": return { action: "fill", ...selector, text: step.text };
case "type": return { action: "type", text: step.text };
case "scroll": return { action: "scroll", direction: step.direction };
case "wait": return { action: "wait", milliseconds: step.milliseconds };
case "open": return { action: "open", url: step.url };
default: return { action: step.action };
}
}
/** As many names as fit in `room` characters, then "and N more"; a count of them when not even the first fits. */
function listed(names: readonly string[], room: number, noun: string): string {
let text = "";
for (const [index, name] of names.entries()) {
const next = text ? `${text}, ${name}` : name;
const left = names.length - index - 1;
if (next.length + (left ? ` and ${left} more`.length : 0) > room) return text ? `${text} and ${names.length - index} more` : `${names.length} ${noun}`;
text = next;
}
return text;
}
const sameTitle = (left: string, right: string): boolean => left.toLowerCase() === right.toLowerCase();
/**
* Runs a fixed plan through the explorer's own tools in place of a model: the same observe, act, and capture calls,
* so the map and report are those of a model run, but every run takes the same steps. A step that fails does not
* fail the run: the report keeps what was captured and names the screens that were not.
*/
export function createReplaySession(plan: ReplayPlan, platform: ReplayPlatform): ReplaySession {
const steps = stepsFor(plan, platform);
const title = (step: ReplayCapture) => step.state ? `${step.capture} (${step.state})` : step.capture;
const missed: string[] = [];
let position = 0;
/** What the plan meant to capture and has not: the screens still ahead first, then the ones earlier steps missed. */
const notCaptured = () => [...steps.slice(position).flatMap(({ step }) => "capture" in step ? [title(step)] : []), ...missed];
return {
provider: "replay",
model: "replay plan",
async preflight() {},
budgetNote() {
const names = notCaptured();
return names.length ? `Not captured: ${listed(names, TOOL_LIMITS.reason, "planned screens")}.` : undefined;
},
async run(request) {
const signal = request.signal ?? new AbortController().signal;
const call = async (name: string, input: Record<string, unknown>) => {
const result = await request.callTool(name, input, signal);
const text = result.content.find(block => block.type === "text");
const value = text && "text" in text ? JSON.parse(text.text) as Record<string, unknown> : {};
return { failed: Boolean(result.isError) || value.ok === false, value };
};
const why = (value: Record<string, unknown>) => String(value.error ?? value.reason ?? value.message ?? "it failed");
// A refused finish or stop_blocked would otherwise end the session with no result, which the explorer reports as a turn budget stop.
const end = async (name: "finish" | "stop_blocked", input: Record<string, unknown>) => {
const ended = await call(name, input);
if (!ended.failed) return;
if (name === "finish") return end("stop_blocked", { reason: `The replay could not finish (${why(ended.value)}). ${String(input.summary)}`.slice(0, TOOL_LIMITS.reason) });
throw new CiError({ code: "REPLAY_STOP_REFUSED", stage: "exploration", message: `The explorer refused the replay's ${name}: ${why(ended.value)}`.slice(0, 600),
nextAction: "Report this with the replay plan; the screens captured so far are in the map." });
};
/** Screen id to the title the plan captured it under. */
const planned = new Map<string, string>();
const skipped: string[] = [];
missed.length = 0;
try {
for (const [index, { step, number }] of steps.entries()) {
position = index;
if ("capture" in step) {
const seen = await call("observe", { includeImage: true });
const frameId = typeof seen.value.frameId === "string" ? seen.value.frameId : undefined;
const known = typeof seen.value.screenId === "string" && !step.state ? planned.get(seen.value.screenId) : undefined;
// The plan already captured this screen: a second capture would only rename it, so it is not made.
if (known !== undefined) {
if (!sameTitle(known, step.capture)) missed.push(`${title(step)} (looked the same as ${known})`);
continue;
}
const captured = frameId && !seen.failed ? await call("capture", { frameId, title: step.capture,
...(step.state ? { state: step.state } : {}), ...(step.description ? { description: step.description } : {}) }) : undefined;
const screenId = typeof captured?.value.screenId === "string" ? captured.value.screenId : undefined;
if (!captured || captured.failed || !screenId) missed.push(title(step));
else if (!planned.has(screenId)) planned.set(screenId, step.capture);
continue;
}
const acted = await call("act", actInput(step));
if (!acted.failed) continue;
if (step.optional) { skipped.push(`step ${number} (${describeAction(step)})`); continue; }
const head = `Replay step ${number} (${describeAction(step)}) failed: ${why(acted.value)}`.slice(0, 600);
const names = notCaptured();
const lead = ". Not captured: ";
await end("stop_blocked", { reason: head + (names.length ? `${lead}${listed(names, TOOL_LIMITS.reason - head.length - lead.length, "planned screens")}` : "") });
return { inputTokens: 0, outputTokens: 0 };
}
position = steps.length;
// The map as it stands: screens merged into others are gone, and each keeps its latest title. One reached but never
// captured is still "Screen N" here and is named after the path to it only once the run ends, so it is counted.
const progress = await call("get_progress", {});
const screens = Array.isArray(progress.value.captured) ? progress.value.captured as Array<{ id?: unknown; title?: unknown }> : [];
const unplanned = screens.filter(screen => typeof screen.id === "string" && !planned.has(screen.id)).map(screen => typeof screen.title === "string" ? screen.title : "");
const named = unplanned.filter(name => name && !isPlaceholderTitle(name));
const unnamed = unplanned.length - named.length;
const reached = [...named, ...(unnamed ? [`${unnamed} unnamed screen${unnamed === 1 ? "" : "s"}`] : [])];
const sections = ([["Not captured", missed, "planned screens"], ["Optional steps skipped because they failed", skipped, "steps"],
["Also reached, not in the plan", reached, "screens"]] as const).filter(([, names]) => names.length);
let summary = `Replayed ${steps.length} planned steps on ${platform}.${missed.length ? "" : " Every planned screen was captured."}`;
for (const [index, [label, names, noun]] of sections.entries()) {
// Each later section keeps room for its label and a count, so a long list of misses does not crowd out the rest.
const reserve = sections.slice(index + 1).reduce((sum, [later]) => sum + later.length + 32, 0);
const lead = ` ${label}: `;
summary += `${lead}${listed(names, TOOL_LIMITS.summary - summary.length - lead.length - 1 - reserve, noun)}.`;
}
await end("finish", { summary });
} catch (error) {
if (signal.aborted) return { inputTokens: 0, outputTokens: 0 };
throw error;
}
return { inputTokens: 0, outputTokens: 0 };
},
};
}
+10 -83
View File
@@ -5,7 +5,6 @@ import { join } from "node:path";
import { createCiAgentSession, type CiAgentSession, type CiAgentSessionConfig } from "../capture/agent/session.js";
import { createLocalCiAgentSession } from "../capture/agent/session-local.js";
import type { CiAgentProviderName } from "../capture/agent/provider.js";
import { assertReplayFits, createReplaySession, parseReplayPlan, replaySetup, type ReplayPlan } from "./replay.js";
import { CONTROLLER_SESSION, baseEnvironment, isRecord } from "../capture/common.js";
import { controllerData, defaultRunner, openedSimulatorDevice } from "../capture/controller.js";
import type { CommandRequest, CommandResult } from "../capture/run.js";
@@ -17,10 +16,8 @@ import { agentDeviceLauncher, ensureArgentToolServer, ensureController } from ".
import { agentDeviceClaims, approveLinkScheme, createLocalSimulator, deleteLocalEmulator, deleteLocalSimulator, hostEmulatorAbi, installOnLocalEmulator, installOnLocalSimulator, startLocalArgentServer,
startLocalEmulator, stopLocalAgentDevice, stopLocalArgentServer, waitForLocalEmulator, waitForLocalSimulator, type LocalArgentServer } from "../capture/local-simulator.js";
import { openLocalCaptureRecord, sweepLocalCaptures } from "../capture/local-registry.js";
import { screenSpace } from "../capture/route-capture.js";
import { approveSessionUrlSchemes, simulatorControlActions, type EasCommand } from "../capture/simulator-control.js";
import { DEFAULT_ANDROID_DEVICE } from "../config/explore-config.js";
import { DUO_PANELS } from "../contracts/capture.js";
import { createExplorationInputs } from "../config/exploration-inputs.js";
import { ExploreStore, getCoverage } from "../map/store.js";
import { whoamiAccounts } from "../report/preview-deploy.js";
@@ -60,8 +57,6 @@ export type BinaryExploreOptions = {
simulator?: "eas" | "local";
environment?: NodeJS.ProcessEnv;
runner?: typeof defaultRunner;
/** Fixed steps to take instead of a model's; no model access is needed. */
replay?: ReplayPlan;
createSession?: (config: CiAgentSessionConfig, options: { env: NodeJS.ProcessEnv; auth: ModelAuth }) => CiAgentSession;
progress?: (message: string) => void;
now?: () => number;
@@ -73,7 +68,7 @@ export type BinaryExploreResult = {
archive: Omit<InspectedArchive, "path"> & { path: string };
session: { id: string; platform: Platform; device: string; owner: string; project: string };
runtime: ActiveSimulatorRuntime;
model: { provider: CiAgentProviderName | "replay"; model: string; auth: "local-login" | "api-key" | "none" };
model: { provider: CiAgentProviderName; model: string; auth: "local-login" | "api-key" };
exploration: BlackBoxExplorerResult;
durationMs: number;
};
@@ -167,8 +162,8 @@ export function resolveSessionApp(runner: typeof defaultRunner, connection: Simu
* App-test inputs: each {{secret:NAME}} in the hints names an environment variable whose value Explore types for the
* model. Read before any session starts, so a missing one costs nothing.
*/
export function hintInputs(hints: string | undefined, environment: NodeJS.ProcessEnv, replay?: ReplayPlan): ReturnType<typeof createExplorationInputs> {
try { return createExplorationInputs({ instructions: hints ?? "", ...(replay ? { setup: replaySetup(replay) } : {}), environment }); }
export function hintInputs(hints: string | undefined, environment: NodeJS.ProcessEnv): ReturnType<typeof createExplorationInputs> {
try { return createExplorationInputs({ instructions: hints ?? "", environment }); }
catch (error) {
if (!(error instanceof CiError) || !error.code.startsWith("EXPLORATION_SECRET_")) throw error;
const name = typeof error.details?.secretName === "string" ? error.details.secretName : "NAME";
@@ -199,36 +194,6 @@ function modelSession(provider: CiAgentProviderName, auth: ModelAuth, options: M
return { session, provider, model: config.model, auth };
}
/** The turns the model gets: three for each action and two for each screen, plus a few to look around and finish. */
const turnBudget = (limits: { maxScreens: number; maxActions: number }): number => limits.maxActions * 3 + limits.maxScreens * 2 + 20;
/**
* A replay plan is checked here as well as by the CLI, for callers that pass one directly. It takes the model's place, so
* the options that only steer a model would do nothing; the plan's own fill and type steps name the secrets it types.
*/
function replayPlan(options: Pick<BinaryExploreOptions, "replay" | "provider" | "model" | "hints">): ReplayPlan | undefined {
if (!options.replay) return undefined;
const ignored = (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
if (ignored.length) {
throw binaryError("REPLAY_OPTIONS_IGNORED", `--replay takes the model's place, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
`Drop ${ignored.map(name => `--${name}`).join(" and ")}. A {{secret:NAME}} in the plan's fill and type steps is read from the environment, as one in --hints is.`);
}
const plan = parseReplayPlan(options.replay);
for (const [index, step] of plan.steps.entries()) {
if (!("action" in step) || step.action !== "open") continue;
try { assertAppUrl(step.url); }
catch (error) { throw new CiError({ code: "REPLAY_PLAN_INVALID", stage: "exploration", message: `Step ${index + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the replay plan." }); }
}
return plan;
}
/** A replay plan takes the model's place: no model access is checked or used. */
function replayModel(plan: ReplayPlan, platform: Platform, limits: { maxScreens: number; maxActions: number }) {
assertReplayFits(plan, platform, { ...limits, maxTurns: turnBudget(limits) });
const session = createReplaySession(plan, platform);
return { session, budgetNote: session.budgetNote, provider: "replay" as const, model: "replay plan", auth: "none" as const };
}
/** Local `claude`/`codex` login unless an API key is present; preflight proves access before a simulator is paid for. */
export function selectModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
const provider: CiAgentProviderName = options.provider ?? (environment.OPENAI_API_KEY?.trim() && !environment.ANTHROPIC_API_KEY?.trim() ? "openai" : "anthropic");
@@ -295,7 +260,7 @@ export type ConnectedSessionExploration = {
limits: { maxDurationMinutes: number; maxScreens: number; maxActions: number };
store: ExploreStore;
map: ExploreMap;
model: { session: Pick<CiAgentSession, "run">; budgetNote?: () => string | undefined };
model: { session: CiAgentSession };
/** `eas simulator:get --json` for the attached session; descriptive (OS version) only. */
sessionJson: () => string;
/** agent-device session name on the daemon; explore:session reuses the caller's session so the device is not "in use" by another. */
@@ -409,20 +374,6 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
catch { return undefined; }
};
const foregroundElsewhere = (): boolean => { const state = foregroundState(); return state ? leftForeground(state, bundleId) : false; };
let unmatchedPanel = false;
/** Whether the frame is an opened Duo's, to be stored turned upright; the run then records the landscape it shows. */
const landscapeOnPortrait = (nodes: readonly Record<string, unknown>[], png: Buffer): boolean => {
const width = png.readUInt32BE(16), height = png.readUInt32BE(20);
const turn = duoScreenshotTurn({ platform, deviceModel: runtime.deviceModel, nodes, width, height });
if (turn === "unmatched" && !unmatchedPanel) {
unmatchedPanel = true;
const root = nodes.find(node => node.type === "Application")?.rect as { width?: unknown; height?: unknown } | undefined;
say(`an iPhone Duo frame (${width}×${height} px, root ${root ? `${String(root.width)}×${String(root.height)} pt` : "missing"}) shows neither of its panels; its screenshot is stored as taken, which may be sideways`);
}
if (turn !== "turn") return false;
if (runtime.orientation !== "landscape") { runtime.orientation = "landscape"; if (map.run.environment) map.run.environment.orientation = "landscape"; }
return true;
};
const observe = async (includeImage: boolean): Promise<BlackBoxFrame> => {
let nodes: Record<string, unknown>[];
let leftApp = false;
@@ -460,7 +411,7 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
assertDecodedPng(png, "Exploration screenshot");
if (png.length > 10 * 1024 * 1024) return { nodes, capturedAt, imageUnavailable: { code: "SCREENSHOT_TOO_LARGE", message: "The screenshot exceeded 10 MB." } };
if (solidColour(png)) return { nodes, capturedAt, imageUnavailable: { code: "SCREENSHOT_BLANK", message: BLANK_SCREENSHOT_MESSAGE }, ...(leftApp ? { leftApp } : {}) };
return { nodes, capturedAt, frameId: randomUUID(), png, imageSha256: createHash("sha256").update(png).digest("hex"), width: png.readUInt32BE(16), height: png.readUInt32BE(20), ...(landscapeOnPortrait(nodes, png) ? { landscape: true } : {}), ...(leftApp ? { leftApp } : {}) };
return { nodes, capturedAt, frameId: randomUUID(), png, imageSha256: createHash("sha256").update(png).digest("hex"), width: png.readUInt32BE(16), height: png.readUInt32BE(20), ...(leftApp ? { leftApp } : {}) };
} finally { await rm(path, { force: true }); }
};
const setupOptions = { command: async (args: string[]) => { const result = controller(guardPositionals([...args, "--json"]), 120_000); return controllerData(result.stdout, args[0] ?? "command"); },
@@ -493,8 +444,7 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
}
const exploration = blockedAtLaunch ?? await runBlackBoxExploration({ session: model.session, store, map, app: { name: app.name, bundleId, platform, ...(app.version ? { version: app.version } : {}) },
inputs, ...(hints ? { hints: hints } : {}), ...(launchUrl ? { launchUrl: launchUrl } : {}), ...(allowDestructive ? { allowDestructive: allowDestructive } : {}), ...(ctx.appIcon ? { appIcon: ctx.appIcon } : {}),
...(model.budgetNote ? { budgetNote: model.budgetNote } : {}),
deadlineMs, maxScreens: limits.maxScreens, maxActions: limits.maxActions, maxTurns: turnBudget(limits), now, observe, progress: say,
deadlineMs, maxScreens: limits.maxScreens, maxActions: limits.maxActions, maxTurns: limits.maxActions * 3 + limits.maxScreens * 2 + 20, now, observe, progress: say,
act: async action => {
const perform = async () => {
if (action.action === "tap" && "frameId" in action) {
@@ -558,8 +508,7 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
const environment = options.environment ?? process.env;
const runner = options.runner ?? defaultRunner;
const limits = validatedLimits(options);
const replay = replayPlan(options);
const inputs = hintInputs(options.hints, environment, replay);
const inputs = hintInputs(options.hints, environment);
if (options.bundleId !== undefined && !appIdPattern.test(options.bundleId)) throw binaryError("BUNDLE_ID_INVALID", `${options.bundleId} is not a bundle or package id.`, "Pass --bundle-id with the app's bundle identifier (iOS) or package name (Android).");
let connection: SimulatorConnection;
try { connection = await resolveAttachedSimulatorSession(environment, options.envFile, options.session); }
@@ -623,7 +572,7 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
map.run.environment = { deviceModel: device };
session = { id: connection.sessionId, platform, device, name: info.name, ...(info.deviceRunSessionUrl ? { url: info.deviceRunSessionUrl } : {}) };
const model = replay ? replayModel(replay, platform, limits) : await preflightModelSession(options, environment, say);
const model = await preflightModelSession(options, environment, say);
const outputDirectory = await realpath(await mkdir(options.output, { recursive: true }).then(() => options.output));
const store = new ExploreStore(outputDirectory);
if (await store.exists()) throw binaryError("OUTPUT_NOT_EMPTY", `${outputDirectory} already holds an Explore map.`, "Pass an empty --output directory.");
@@ -716,8 +665,7 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
const environment = options.environment ?? process.env;
const runner = options.runner ?? defaultRunner;
const limits = validatedLimits(options);
const replay = replayPlan(options);
const inputs = hintInputs(options.hints, environment, replay);
const inputs = hintInputs(options.hints, environment);
const local = options.simulator === "local";
// A local emulator runs this Mac's ABI (arm64-v8a on Apple silicon); EAS's runs x86_64.
const emulatorAbi = local ? hostEmulatorAbi() : "x86_64";
@@ -734,7 +682,7 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
say(`${archive.appName} ${archive.version ?? ""} (${bundleId}, ${platform}, ${(archive.bytes / 1048576).toFixed(0)} MB, sha256 ${archive.sha256.slice(0, 12)})`);
if (archive.missingLibs?.length && emulatorAbi === "x86_64") say(`warning: lib/x86_64 lacks ${archive.missingLibs.slice(0, 4).join(", ")}${archive.missingLibs.length > 4 ? ` and ${archive.missingLibs.length - 4} more` : ""} that another ABI ships; the app may stop with UnsatisfiedLinkError on the emulator`);
const model = replay ? replayModel(replay, platform, limits) : await preflightModelSession(options, environment, say);
const model = await preflightModelSession(options, environment, say);
const outputDirectory = await realpath(await mkdir(options.output, { recursive: true }).then(() => options.output));
const store = new ExploreStore(outputDirectory);
@@ -923,27 +871,6 @@ export function guardPositionals(args: readonly string[]): string[] {
const KEYBOARD_RETURN_KEY = /^(return|next|done|go|continue|join)$/i;
/**
* How a frame's screenshot is stored. An opened (or half-open) iPhone Duo lights its inner panel, whose native axes are
* portrait, and shows a landscape UI on it: the tree and taps stay on the portrait axes (a 669×951 pt root, the transpose
* of `DUO_PANELS.inner`), and a screenshot on those axes is sideways to a person, so it is turned ("turn"). Local
* agent-device runs returned exactly that (2007×2853 px). Capture's Argent screenshots on EAS, taken with Argent's
* LandscapeRight rotation, came back upright (2853×2007); an image that is already the root's transpose is left as it is.
* The root is the full-screen Application node capture takes its coordinate space from, matched to within a pixel and a
* point. A Duo frame on neither panel is "unmatched": it is stored as taken, and the caller says so. An empty tree, read
* mid-draw, has nothing to judge.
*/
export function duoScreenshotTurn(frame: { platform: Platform; deviceModel: string; nodes: readonly Record<string, unknown>[]; width: number; height: number }): "turn" | "keep" | "unmatched" {
if (frame.platform !== "ios" || !/\bduo\b/i.test(frame.deviceModel) || !frame.nodes.length) return "keep";
const space = screenSpace(frame.nodes, frame, 1);
if (!space) return "unmatched";
const near = (value: number, points: number) => Math.abs(value - points) < 1;
const portraitAxes = near(space.width, DUO_PANELS.inner.height) && near(space.height, DUO_PANELS.inner.width);
if (portraitAxes && !space.transposed && frame.width < frame.height) return "turn";
const panel = Object.values(DUO_PANELS).some(({ width, height }) => (near(space.width, width) && near(space.height, height)) || (near(space.width, height) && near(space.height, width)));
return panel ? "keep" : "unmatched";
}
/** A node that is not the app's UI: another package drew it (Android's system UI over the app), or it is Explore's own route marker, which an app
* instrumented for PR runs renders as a 2x2 hidden text. */
export function foreignNode(node: Record<string, unknown>, bundleId: string): boolean {
+10 -21
View File
@@ -112,24 +112,6 @@ export function verifiedCaptureLabels(nodes: readonly Record<string, unknown>[],
}));
}
/**
* The full-screen logical space a screenshot shows: the one Application node at the origin whose aspect is the image's,
* at any pixel density. Failing that, the one whose aspect is the image's transpose (`transposed`): the image is then the
* screen turned a quarter. A square screen matches both ways and is read upright. `slack` is how many image pixels the
* root, scaled to the image, may miss it by; capture asks for an exact match.
*/
export function screenSpace(nodes: readonly Record<string, unknown>[], image: { width: number; height: number }, slack = 0): { width: number; height: number; transposed: boolean } | undefined {
const roots = (transposed: boolean) => nodes.filter(node => node.type === "Application" && isRecord(node.rect)
&& node.rect.x === 0 && node.rect.y === 0 && typeof node.rect.width === "number" && typeof node.rect.height === "number"
&& Number.isFinite(node.rect.width) && Number.isFinite(node.rect.height) && node.rect.width > 0 && node.rect.height > 0
&& Math.abs(transposed ? node.rect.width * image.width - node.rect.height * image.height : node.rect.width * image.height - node.rect.height * image.width) <= slack * node.rect.height);
const upright = roots(false), turned = upright.length ? [] : roots(true);
const matched = upright.length ? upright : turned;
if (matched.length !== 1) return undefined;
const rect = matched[0]!.rect as { width: number; height: number };
return { width: rect.width, height: rect.height, transposed: turned.length === 1 };
}
/**
* A swipe that closes a presented sheet, or undefined when the snapshot does not prove a sheet.
* The route observer is drawn at the focused scene's top-left corner. A sheet's scene starts below
@@ -748,11 +730,18 @@ export async function capturePlannedRoutes(options: {
* then the transpose of the frame, and a tap is turned back (see `toLogical`).
*/
const bounds = (frame: Candidate) => {
const roots = (transposed: boolean) => frame.snapshot.nodes.filter(node => node.type === "Application" && isRecord(node.rect)
&& node.rect.x === 0 && node.rect.y === 0 && typeof node.rect.width === "number" && typeof node.rect.height === "number"
&& Number.isFinite(node.rect.width) && Number.isFinite(node.rect.height) && node.rect.width > 0 && node.rect.height > 0
&& (transposed ? node.rect.width * frame.width === node.rect.height * frame.height : node.rect.width * frame.height === node.rect.height * frame.width));
// A square screen matches both ways; it is read upright, as before rotated frames existed.
const upright = roots(false), turned = upright.length ? [] : roots(true);
const matched = upright.length ? upright : turned;
if (matched.length !== 1) throw captureError("AGENT_COORDINATE_BOUNDS_UNVERIFIED", "The screenshot does not identify one matching full-screen logical coordinate space.");
// The official iOS screenshot is full-screen. Its default is 1x, but a proven
// uniform pixel density must still map to XCTest points rather than raw pixels.
const space = screenSpace(frame.snapshot.nodes, frame);
if (!space) throw captureError("AGENT_COORDINATE_BOUNDS_UNVERIFIED", "The screenshot does not identify one matching full-screen logical coordinate space.");
return space;
const rect = matched[0]!.rect as { width: number; height: number };
return { width: rect.width, height: rect.height, transposed: turned.length === 1 };
};
/**
* A normalized point (x, y) in the screenshot, in the tree's logical points. Upright, it scales by the root. Turned:
-3
View File
@@ -825,9 +825,6 @@ async function executeCiMap(options: CiMapOptions, setStage: (stage: string) =>
// can leave the panel it lights black, as it does for Argent: the controller then reads the old panel and a
// relaunch draws nothing. The display is the check, through the controller's own screenshot: in the pose's shape
// and drawn. If it is not, SpringBoard is restarted, the app reopened and the device folded again, once.
// The open pose's shape is taken to be landscape. That is unverified for agent-device on EAS: local agent-device
// runs returned the opened inner panel on its portrait axes (2007×2853), as explore:binary turns it from; the
// landscape shots seen on EAS (2853×2007) were Argent's, taken with its LandscapeRight rotation.
const open = (relaunch: boolean) => runner({ command: controllerCommand, args: ["open", options.bundleId, ...(relaunch ? ["--relaunch"] : []), "--session", CONTROLLER_SESSION,
...target, "--platform", options.platform, "--json"], cwd: simulatorProject, environment: controllerEnvironment, timeoutMs: 15 * 60_000 });
const shown = async () => {
+5 -15
View File
@@ -100,24 +100,13 @@ function failure(code: string, message: string, details?: Record<string, unknown
...(details ? { details } : {}) });
}
export const SETUP_SELECTOR_MAX = 256;
/** A label or identifier the executor can select by; a replay plan's selectors are checked against it before any device starts. */
export function isSetupSelector(value: unknown): value is string {
return typeof value === "string" && value.trim().length > 0 && value.length <= SETUP_SELECTOR_MAX && !/[\u0000-\u001f\u007f]/.test(value);
}
/** Text the executor can fill or type: one line of at most 4,096 characters. */
export function isSetupInput(value: unknown): value is string {
return typeof value === "string" && value.length <= 4_096 && !/[\u0000\r\n]/.test(value);
}
function selectorValue(selector: CaptureSetupSelector): { key: "label" | "identifier"; value: string } {
const hasLabel = typeof selector.label === "string";
const hasIdentifier = typeof selector.identifier === "string";
const key = hasLabel ? "label" : "identifier";
const value = selector[key];
if (hasLabel === hasIdentifier || !isSetupSelector(value)) {
if (hasLabel === hasIdentifier || typeof value !== "string" || !value.trim()
|| value.length > 256 || /[\u0000-\u001f\u007f]/.test(value)) {
throw failure("SETUP_SELECTOR_INVALID", "A UI selector must contain exactly one nonempty label or identifier.");
}
return { key, value };
@@ -287,7 +276,8 @@ export async function executeCaptureSetup(actions: readonly CaptureSetupAction[]
case "tap":
case "fill": {
selectorValue(action);
if (action.action === "fill" && !isSetupInput(action.text)) {
if (action.action === "fill" && (typeof action.text !== "string" || action.text.length > 4_096
|| /[\u0000\r\n]/.test(action.text))) {
throw failure("SETUP_INPUT_INVALID", "A UI fill input must be a single line of at most 4,096 characters.");
}
let resolved = false;
@@ -342,7 +332,7 @@ export async function executeCaptureSetup(actions: readonly CaptureSetupAction[]
break;
}
case "type": {
if (!isSetupInput(action.text) || !action.text) {
if (typeof action.text !== "string" || !action.text || action.text.length > 4_096 || /[\u0000\r\n]/.test(action.text)) {
throw failure("SETUP_INPUT_INVALID", "A UI type input must be a nonempty single line of at most 4,096 characters.");
}
let ready = false;
+2 -2
View File
@@ -90,10 +90,10 @@ const flagKinds: Record<string, Record<string, "value" | "boolean">> = {
"explore:binary": { replay:"value", archive:"value", output:"value", platform:"value", controller:"value", simulator:"value", device:"value", "bundle-id":"value", "launch-url":"value", hints:"value", "allow-destructive":"value",
"explore:binary": { archive:"value", output:"value", platform:"value", controller:"value", simulator:"value", device:"value", "bundle-id":"value", "launch-url":"value", hints:"value", "allow-destructive":"value",
"max-duration-minutes":"value", "max-screens":"value", "max-actions":"value", provider:"value", model:"value", owner:"value", project:"value",
upload:"boolean", "no-upload":"boolean", "expires-in":"value", alias:"value", "no-open":"boolean", port:"value" },
"explore:session": { replay:"value", session:"value", "env-file":"value", "bundle-id":"value", "app-name":"value", "app-version":"value", output:"value", platform:"value", controller:"value", device:"value", "launch-url":"value", hints:"value",
"explore:session": { session:"value", "env-file":"value", "bundle-id":"value", "app-name":"value", "app-version":"value", output:"value", platform:"value", controller:"value", device:"value", "launch-url":"value", hints:"value",
"allow-destructive":"value", "max-duration-minutes":"value", "max-screens":"value", "max-actions":"value", provider:"value", model:"value", owner:"value", project:"value",
upload:"boolean", "expires-in":"value", alias:"value", "no-open":"boolean", port:"value" },
"ci:report": { input:"value", output:"value", target:"value", platform:"value" },
-3
View File
@@ -7,7 +7,6 @@ import { assertHostingExport, deployExplorationReport, zipReportDirectory } from
import { keepReports } from "./report-commands.js";
import { warnIgnoredExpiry } from "../config/explore-config.js";
import { argumentError, flag, flags, has, nonNegativeInteger, oneOf, output, positional, positiveInteger } from "./args.js";
import { readReplayPlan } from "../binary/replay.js";
/** `--alias NAME` plus a fixed token from EXPLORE_PREVIEW_TOKEN: the hosted link stays the same from one run to the next (validated by the deploy). */
@@ -50,7 +49,6 @@ export async function exploreBinary(): Promise<void> {
...(flag("max-duration-minutes") ? { maxDurationMinutes: positiveInteger("max-duration-minutes") } : {}),
...(flag("max-screens") ? { maxScreens: positiveInteger("max-screens") } : {}),
...(flag("max-actions") ? { maxActions: positiveInteger("max-actions") } : {}),
...(flag("replay") ? { replay: await readReplayPlan(resolve(process.cwd(), flag("replay")!)) } : {}),
});
say(`exploration ${result.exploration.stopReason}: ${result.exploration.screens} screens, ${result.exploration.observations} captures, ${result.exploration.actions} actions`);
const report = await writeExplorationReport(result, join(outputDirectory, "report"));
@@ -115,7 +113,6 @@ export async function exploreSession(): Promise<void> {
...(flag("max-duration-minutes") ? { maxDurationMinutes: positiveInteger("max-duration-minutes") } : {}),
...(flag("max-screens") ? { maxScreens: positiveInteger("max-screens") } : {}),
...(flag("max-actions") ? { maxActions: positiveInteger("max-actions") } : {}),
...(flag("replay") ? { replay: await readReplayPlan(resolve(process.cwd(), flag("replay")!)) } : {}),
});
say(`exploration ${result.exploration.stopReason}: ${result.exploration.screens} screens, ${result.exploration.observations} captures, ${result.exploration.actions} actions`);
const report = await writeExplorationReport(result, join(outputDirectory, "report"));
-1
View File
@@ -56,7 +56,6 @@ const BINARY = `Prebuilt apps (no source needed; runs with your local claude or
and the model and the report only see the reference)
--max-duration-minutes N --max-screens N --max-actions N --provider anthropic|openai --model ID
--owner ACCOUNT --project SLUG --allow-destructive "Label,Other label"
--replay FILE (takes the steps in FILE instead of a model's, so every run and device takes the same path; no model access needed, and not with --provider, --model or --hints)
--alias NAME (one EAS Hosting host per app that always serves the newest run; with EXPLORE_PREVIEW_TOKEN
set to a fixed 32-character token, the hosted link is the same every run)
(iOS needs a simulator build; a device .ipa and Android .aab are rejected before any device starts)
+1 -1
View File
@@ -178,7 +178,7 @@ export type SystemRegion = { kind: "status-bar" | "home-indicator"; x: number; y
* EAS Simulator captures of both panels; the pill is the same 40×64 points on each, only its top differs. The app's own
* rail controls start about 3% of the height below it on the outer panel, so the padding stays small.
*/
export const DUO_PANELS = {
const DUO_PANELS = {
outer: { width: 466, height: 678, statusBar: { x: 398, y: 87, width: 40, height: 64 } },
inner: { width: 951, height: 669, statusBar: { x: 883, y: 34, width: 40, height: 64 } },
} as const;
+8 -23
View File
@@ -254,39 +254,24 @@ function explorationScreenItem(context: AgentContext, map: Json, screen: Json):
viewerUrl: ID.test(id) ? viewerUrl(context, { screen: id }) : undefined,
});
}
/** The screenshot a screen is shown by: its latest observation's. */
function screenshotArtifact(map: Json, screen: Json): Json | undefined {
function screenshotUrl(context: AgentContext, map: Json, screen: Json): string | undefined {
const observationId = strings(screen.observationIds).at(-1);
const observation = arr(map.observations).map(rec).find(o => o.id === observationId);
return arr(map.artifacts).map(rec).find(a => a.id === observation?.screenshotArtifactId);
}
function screenshotUrl(context: AgentContext, map: Json, screen: Json): string | undefined {
const sha = str(screenshotArtifact(map, screen)?.sha256);
const artifact = arr(map.artifacts).map(rec).find(a => a.id === observation?.screenshotArtifactId);
const sha = str(artifact?.sha256);
return sha && SHA256.test(sha) ? withPlatform(context, apiUrl(context, `/assets/${sha}`)) : undefined;
}
/**
* One element, its bounds on the axes of the screenshot it is listed beside. A screenshot stored turned a quarter
* clockwise (an opened iPhone Duo's) is `turnedHeight` points tall on the tree's axes, where the map keeps the bounds:
* there the portrait point (x, y) shows at (H - y, x).
*/
const elementItem = (element: Json, turnedHeight?: number): Json => {
const bounds = element.bounds === undefined ? undefined : rec(element.bounds);
const [x, y, width, height] = [num(bounds?.x), num(bounds?.y), num(bounds?.width), num(bounds?.height)];
const shown = bounds && turnedHeight !== undefined && bounds.unit === "points" && x !== undefined && y !== undefined && width !== undefined && height !== undefined
? { x: turnedHeight - y - height, y: x, width: height, height: width, unit: "points" } : bounds;
return defined({
id: str(element.id ?? element.elementId), label: str(element.label), role: str(element.role), value: str(element.value), identifier: str(element.identifier),
bounds: shown, actions: strings(element.actions), risk: str(element.risk), actionable: bool(element.actionable), enabled: bool(element.enabled), visible: bool(element.visible),
});
};
const elementItem = (element: Json): Json => defined({
id: str(element.id ?? element.elementId), label: str(element.label), role: str(element.role), value: str(element.value), identifier: str(element.identifier),
bounds: element.bounds === undefined ? undefined : rec(element.bounds), actions: strings(element.actions), risk: str(element.risk), actionable: bool(element.actionable), enabled: bool(element.enabled), visible: bool(element.visible),
});
const transitionItem = (transition: Json): Json => defined({
id: str(transition.id), from: str(transition.fromScreenId), to: str(transition.toScreenId), actionKind: str(transition.actionKind), elementId: str(transition.elementId), attempts: strings(transition.attemptIds).length,
});
function explorationScreenDetail(context: AgentContext, map: Json, screen: Json): Json {
const id = str(screen.id);
const transitions = arr(map.transitions).map(rec);
const turnedHeight = num(rec(screenshotArtifact(map, screen)?.turned).height);
const elements = arr(map.elements).map(rec).filter(element => element.screenId === id).map(element => elementItem(element, turnedHeight));
const elements = arr(map.elements).map(rec).filter(element => element.screenId === id).map(elementItem);
const variants = arr(map.variants).map(rec).filter(variant => variant.screenId === id).map(variant => defined({ id: str(variant.id), key: str(variant.key), name: str(variant.name), observations: strings(variant.observationIds).length }));
const gaps = arr(map.gaps).map(rec).filter(gap => gap.screenId === id).map(gapItem);
return defined({
-36
View File
@@ -144,39 +144,3 @@ export function assertDecodedPng(contents: Buffer, label: string): void {
if (decoded[row * (rowBytes + 1)]! > 4) throw new Error(`${label} has an invalid PNG row filter`);
}
}
/** The chunks that say how a PNG's samples map to colour; pngjs reads past them and writes none of its own. */
const COLOUR_CHUNKS = new Set(["iCCP", "sRGB", "gAMA", "cHRM"]);
/**
* The image turned a quarter clockwise. An opened iPhone Duo shows a landscape UI on a panel whose native axes are
* portrait, and its screenshot comes back on those axes; turned, it is what a person sees. The source's colour chunks
* are carried over unchanged. pngjs reads every PNG as 8-bit RGBA, so a 16-bit image comes out 8-bit; simulator
* screenshots are 8-bit already.
*/
export function turnedClockwise(png: Buffer): Buffer {
const { PNG } = createRequire(import.meta.url.replace(/\?.*$/, ""))("pngjs") as { PNG: { new(size: { width: number; height: number }): DecodedFrame; sync: {
read(bytes: Buffer, options: { checkCRC: boolean }): DecodedFrame; write(frame: DecodedFrame): Buffer } } };
const source = PNG.sync.read(png, { checkCRC: false });
const { width, height } = source;
const turned = new PNG({ width: height, height: width });
for (let y = 0; y < height; y += 1) {
for (let x = 0; x < width; x += 1) {
const from = (y * width + x) * 4;
source.data.copy(turned.data, (x * height + (height - 1 - y)) * 4, from, from + 4);
}
}
const written = PNG.sync.write(turned);
// Each chunk is copied whole, CRC included: its data does not depend on the pixels or their size. They belong before
// the image data, so they go right after the header (8 bytes of signature, then IHDR's 25). The output is RGBA, so a
// grey source's ICC profile, which describes grey samples only, is left out.
const grey = png[25] === 0 || png[25] === 4;
const colour: Buffer[] = [];
for (let offset = 8; offset + 12 <= png.length;) {
const length = png.readUInt32BE(offset), type = png.toString("latin1", offset + 4, offset + 8), next = offset + 12 + length;
if (type === "IDAT" || type === "IEND" || next > png.length) break;
if (COLOUR_CHUNKS.has(type) && !(grey && type === "iCCP")) colour.push(png.subarray(offset, next));
offset = next;
}
return colour.length ? Buffer.concat([written.subarray(0, 33), ...colour, written.subarray(33)]) : written;
}
-6
View File
@@ -121,12 +121,6 @@ export type Artifact = {
mimeType: string;
width?: number;
height?: number;
/**
* A screenshot stored turned a quarter clockwise from its accessibility tree's axes (an opened iPhone Duo's landscape UI
* on its inner panel's portrait axes): the tree's screen, `width` × `height` points. Element bounds stay on the tree's
* axes, where the portrait point (x, y) shows at (height - y, x).
*/
turned?: { width: number; height: number };
createdAt: string;
};
+1 -327
View File
@@ -8,8 +8,7 @@ import { tsImport } from "tsx/esm/api";
const { runBlackBoxExploration, screenEvidenceKey, elementsOf, fieldName, systemDialogTitle } = await tsImport("../../src/binary/explorer.ts", import.meta.url);
const { ExploreStore, validateMap } = await tsImport("../../src/map/store.ts", import.meta.url);
const { CONTRACT } = await tsImport("../../src/types.ts", import.meta.url);
const tsRun = await tsImport("../../src/binary/run.ts", import.meta.url);
const { assertAppUrl, describeForeground, foreignNode, guardPositionals, leftForeground, retryWhileBusy } = tsRun;
const { assertAppUrl, describeForeground, foreignNode, guardPositionals, leftForeground, retryWhileBusy } = await tsImport("../../src/binary/run.ts", import.meta.url);
const PNG = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==", "base64");
const SCREENS = {
@@ -882,328 +881,3 @@ test("when nothing passes preflight, the first failure is reported", async () =>
const createSession = (config, { auth }) => ({ async preflight() { throw new Error(`${config.provider} ${auth} refused`); } });
await assert.rejects(preflightModelSession({ createSession }, {}, () => {}), /anthropic local-login refused/);
});
test("a landscape frame on a portrait panel (an opened iPhone Duo) is stored upright", async () => {
const { createRequire } = await import("node:module");
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
const raw = new Png({ width: 1, height: 2 });
raw.data.set([255, 0, 0, 255, 0, 0, 255, 255]);
const png = Png.sync.write(raw);
const { map, root } = await harness([["observe", { includeImage: true }], ["capture", { frameId: "frame-0", title: "Home" }], ["finish", { summary: "One." }]], {
observe: async includeImage => ({ nodes: SCREENS.home, capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png, width: 1, height: 2, landscape: true } : {}) }) });
const artifact = map.artifacts.find(item => item.kind === "screenshot");
assert.deepEqual([artifact.width, artifact.height], [2, 1]);
const { readFile } = await import("node:fs/promises");
const stored = Png.sync.read(await readFile(join(root, artifact.path)));
// The panel's top (red) is the UI's right-hand side once turned a quarter clockwise.
assert.deepEqual([...stored.data], [0, 0, 255, 255, 255, 0, 0, 255]);
});
/** A PNG of `width` × `height` RGBA pixels, each given as [r, g, b, a] row by row. */
async function pngOf(width, height, pixels) {
const { createRequire } = await import("node:module");
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
const image = new Png({ width, height });
image.data.set(pixels.flat());
return Png.sync.write(image);
}
test("an opened Duo's frame shows the same pixels as a screen already seen once turned, so it is that screen (AudioScape's playlist title, rotated)", async () => {
const tabs = [{ type: "Button", label: "Home", hittable: true }, { type: "Button", label: "Library", hittable: true }];
const library = [{ type: "StaticText", label: "Library" }, { type: "Button", label: "My Playlist", hittable: true }, ...tabs];
const trees = { library, titled: [{ type: "StaticText", label: "My Playlist" }, ...tabs], untitled: [...tabs] };
const images = { library: await pngOf(1, 2, [[255, 0, 0, 255], [0, 0, 255, 255]]), playlist: await pngOf(1, 2, [[0, 255, 0, 255], [0, 0, 255, 255]]) };
let visits = 0; let frames = 0; let where = "library";
const { map } = await harness([
["observe", { includeImage: true }], ["capture", { frameId: "frame-0", title: "Library" }],
["act", { action: "tap", label: "My Playlist" }], ["observe", { includeImage: true }],
["act", { action: "back" }], ["observe", { includeImage: true }],
["act", { action: "tap", label: "My Playlist" }], ["observe", { includeImage: true }],
["finish", { summary: "Done." }],
], { observe: async includeImage => ({ nodes: trees[where], capturedAt: new Date().toISOString(),
...(includeImage ? { frameId: `frame-${frames++}`, png: Buffer.from(images[where === "library" ? "library" : "playlist"]), width: 1, height: 2, landscape: true } : {}) }),
act: async action => { if (action.action === "back") where = "library"; else where = visits++ === 0 ? "titled" : "untitled"; } });
assert.equal(map.screens.length, 2, "the second read of the playlist did not mint a screen");
assert.equal(map.observations.filter(item => item.screenId === map.screens[1].id).length, 2, "both reads are observations of the one playlist screen");
assert.equal(map.artifacts.filter(item => item.kind === "screenshot").length, 2);
});
test("a coordinate tap on an opened Duo is named by where it shows in the upright screenshot", async () => {
// The inner panel's portrait axes: a button near the portrait top right is at the upright bottom right.
const home = [
{ type: "Application", label: "", rect: { x: 0, y: 0, width: 669, height: 951 } },
{ type: "Button", label: "", hittable: true, rect: { x: 600, y: 40, width: 44, height: 44 } },
];
let frames = 0; let tapped = false;
// The look without an image carries no landscape flag; the screen's stored screenshot still says it was turned.
const { map } = await harness([
["observe", { includeImage: true }], ["capture", { frameId: "frame-0", title: "Home" }], ["observe", { includeImage: false }],
["act", { action: "tap", frameId: "frame-0", x: 622 / 669, y: 62 / 951 }],
["finish", { summary: "Done." }],
], { observe: async includeImage => ({ nodes: tapped ? SCREENS.settings : home, capturedAt: new Date().toISOString(),
...(includeImage ? { frameId: `frame-${frames++}`, png: await pngOf(1, 2, [[255, 0, 0, 255], [0, 0, 255, 255]]), width: 1, height: 2, landscape: true } : {}) }),
act: async () => { tapped = true; } });
assert.equal(map.actionAttempts.find(item => item.action.kind === "press").action.label, "icon button, bottom right");
assert.deepEqual(map.artifacts.find(item => item.kind === "screenshot").turned, { width: 669, height: 951 });
});
test("turnedClockwise puts the portrait top on the right and keeps the colour chunks", async () => {
const { turnedClockwise, assertDecodedPng } = await tsImport("../../src/shared/png-evidence.ts", import.meta.url);
const { createRequire } = await import("node:module");
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
const { crc32 } = await import("node:zlib");
// 3 wide and 2 tall; each pixel's red is its index, so where it lands says where it came from.
const plain = await pngOf(3, 2, [0, 1, 2, 3, 4, 5].map(index => [index * 40, 0, 0, 255]));
const chunk = (type, data) => {
const body = Buffer.concat([Buffer.from(type, "latin1"), data]), out = Buffer.alloc(body.length + 8);
out.writeUInt32BE(data.length, 0); body.copy(out, 4); out.writeUInt32BE(crc32(body), body.length + 4);
return out;
};
const gamma = Buffer.alloc(4); gamma.writeUInt32BE(45455);
const source = Buffer.concat([plain.subarray(0, 33), chunk("sRGB", Buffer.from([0])), chunk("gAMA", gamma), plain.subarray(33)]);
const turned = turnedClockwise(source);
assertDecodedPng(turned, "Turned");
const decoded = Png.sync.read(turned);
assert.deepEqual([decoded.width, decoded.height], [2, 3]);
// Source (x, y) lands at (height - 1 - y, x): the top row (0, 1, 2) runs down the right-hand column.
const red = [...Array(6).keys()].map(index => decoded.data[index * 4] / 40);
assert.deepEqual(red, [3, 0, 4, 1, 5, 2]);
const types = [];
for (let offset = 8; offset < turned.length; offset += 12 + turned.readUInt32BE(offset)) types.push(turned.toString("latin1", offset + 4, offset + 8));
assert.deepEqual(types.slice(0, 3), ["IHDR", "sRGB", "gAMA"]);
assert.ok(turned.includes(gamma), "the gamma is the source's");
});
test("only an opened Duo's frame on its inner panel's portrait axes is turned upright", () => {
const { duoScreenshotTurn } = tsRun;
const root = (width, height) => ({ type: "Application", label: "Expo Go", rect: { x: 0, y: 0, width, height } });
const frame = (deviceModel, nodes, width, height, platform = "ios") => duoScreenshotTurn({ platform, deviceModel, nodes, width, height });
const cases = [
["Duo open", frame("iPhone Duo", [root(669, 951)], 2007, 2853), "turn"],
["Duo half-open", frame("iPhone Duo", [root(669, 951), { type: "Button", label: "Home", rect: { x: 20, y: 40, width: 44, height: 44 } }], 2007, 2853), "turn"],
["Duo closed", frame("iPhone Duo", [root(466, 678)], 1398, 2034), "keep"],
["iPhone 17 Pro", frame("iPhone 17 Pro", [root(402, 874)], 1206, 2622), "keep"],
["an iPhone in landscape", frame("iPhone 17 Pro", [root(874, 402)], 2622, 1206), "keep"],
["iPad", frame("iPad Pro 13-inch (M5)", [root(1032, 1376)], 2064, 2752), "keep"],
["Android", frame("Pixel 9", [root(669, 951)], 2007, 2853, "android"), "keep"],
["an empty tree", frame("iPhone Duo", [], 2007, 2853), "keep"],
["a first node that isn't the Application", frame("iPhone Duo", [{ type: "Window", rect: { x: 0, y: 0, width: 300, height: 200 } }, root(669, 951)], 2007, 2853), "turn"],
["a root off by a fraction of a point", frame("iPhone Duo", [root(669.33, 951)], 2008, 2853), "turn"],
["an already-landscape image", frame("iPhone Duo", [root(669, 951)], 2853, 2007), "keep"],
["the inner panel on its own axes", frame("iPhone Duo", [root(951, 669)], 2853, 2007), "keep"],
["a Duo root on neither panel", frame("iPhone Duo", [root(700, 1000)], 2100, 3000), "unmatched"],
["a Duo root the image does not show", frame("iPhone Duo", [root(669, 951)], 2000, 2000), "unmatched"],
];
for (const [name, actual, expected] of cases) assert.equal(actual, expected, name);
});
test("the agent API lists a turned screenshot's element bounds on its upright axes", async () => {
const { agentApiReply } = await tsImport("../../src/report/agent-api.ts", import.meta.url);
const sha = "a".repeat(64);
const map = { screens: [{ id: "scr_home", title: "Home", observationIds: ["obs_1"] }], observations: [{ id: "obs_1", screenshotArtifactId: "art_1" }],
artifacts: [{ id: "art_1", kind: "screenshot", sha256: sha, width: 2853, height: 2007, turned: { width: 669, height: 951 } }],
elements: [{ id: "el_1", screenId: "scr_home", label: "Settings", role: "button", bounds: { x: 600, y: 40, width: 44, height: 30, unit: "points" } }], transitions: [], variants: [], gaps: [] };
const loaded = { kind: "exploration", file: "exploration.json", raw: new Uint8Array(), json: { map }, entries: new Map(), read: async () => undefined };
const reply = await agentApiReply({ pageUrl: "https://example.test/p/token", platforms: ["ios"], expiresAt: null }, loaded, "/screens/scr_home", new URLSearchParams(), "application/json");
const body = JSON.parse(await new Response(reply.body).text());
// Portrait (600, 40) on a 951-point-tall screen shows at (951 - 40 - 30, 600) upright, near the bottom right.
assert.deepEqual(body.elements[0].bounds, { x: 881, y: 600, width: 30, height: 44, unit: "points" });
});
const { assertReplayFits, createReplaySession, parseReplayPlan, replaySetup } = await tsImport("../../src/binary/replay.ts", import.meta.url);
function replayDevice(screens = SCREENS) {
let screen = "home", frames = 0;
const acted = [];
return { acted, options: {
observe: async includeImage => ({ nodes: screens[screen], capturedAt: new Date().toISOString(),
...(includeImage ? { frameId: `replay-${++frames}`, png: PNG, imageSha256: `${frames}`.padStart(64, "b") } : {}) }),
act: async action => {
acted.push(action);
if (action.label === "Missing" || action.label === "Nowhere") throw new Error(`No element labelled ${action.label}`);
if (action.action === "tap" && action.label === "Settings") screen = "settings";
if (action.action === "tap" && action.label === "Dark mode" && screens.settingsDark) screen = "settingsDark";
if (action.action === "back") screen = "home";
},
} };
}
test("a replay plan takes the same steps on each run: platform-only steps are skipped, optional failures are passed over and listed", async () => {
const plan = parseReplayPlan({ steps: [
{ capture: "Home" },
{ action: "tap", label: "Missing", optional: true },
{ action: "tap", label: "Android only", platforms: ["android"] },
{ action: "tap", label: "Settings" }, { capture: "Settings" },
{ action: "tap", label: "Dark mode" }, { capture: "Settings", state: "dark" },
{ action: "back" },
] });
const device = replayDevice();
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
assert.equal(result.stopReason, "finished");
assert.match(result.summary, /Every planned screen was captured\. Optional steps skipped because they failed: step 2 \(tap "Missing"\)\.$/);
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Settings"]);
assert.ok(!device.acted.some(action => action.label === "Android only"));
assert.ok(map.variants.some(variant => variant.name === "dark"), "the dark state is recorded as a variant");
});
test("a failed replay step stops the run, keeps what was captured, and names the screens it did not reach", async () => {
const plan = parseReplayPlan({ steps: [
{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" }, { action: "back" },
{ action: "tap", label: "Nowhere" }, { capture: "Never" }, { capture: "Android screen", platforms: ["android"] },
] });
const device = replayDevice();
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
assert.equal(result.stopReason, "blocked");
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Settings"]);
assert.match(result.summary, /Replay step 5 \(tap "Nowhere"\) failed: .*No element labelled Nowhere.*Not captured: Never$/);
});
test("a blocked step names the screens still ahead before earlier misses, within stop_blocked's limit", async () => {
const plan = parseReplayPlan({ steps: [
{ capture: "Home" },
...Array.from({ length: 20 }, (_, index) => ({ capture: `Earlier miss ${index} ${"x".repeat(60)}` })),
{ action: "tap", label: "Nowhere" },
{ capture: "Ahead" },
] });
const device = replayDevice();
const { result } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options, maxTurns: 100 });
assert.equal(result.stopReason, "blocked");
assert.ok(result.summary.length <= 1000, `${result.summary.length} characters`);
assert.match(result.summary, /Not captured: Ahead, Earlier miss 0 .* and \d+ more$/);
});
test("a destructive or app-leaving tap in a replay plan is refused, as a model's would be", async () => {
const plan = parseReplayPlan({ steps: [
{ capture: "Home" }, { action: "tap", label: "Share", optional: true }, { action: "tap", label: "Delete account" }, { capture: "Gone" },
] });
const device = replayDevice();
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
assert.equal(result.stopReason, "blocked");
assert.match(result.summary, /^Replay step 3 \(tap "Delete account"\) failed: "Delete account" looks destructive.*Not captured: Gone$/);
assert.deepEqual(device.acted, [], "neither tap reaches the device");
assert.deepEqual(map.actionAttempts.map(attempt => [attempt.action.label, attempt.outcome]), [["Share", "blocked"], ["Delete account", "blocked"]]);
});
test("a replay plan is checked before any device starts: unknown, hostile and malformed fields are refused", () => {
const refused = (step, pattern) => assert.throws(() => parseReplayPlan({ steps: [step] }), error => error.code === "REPLAY_PLAN_INVALID" && pattern.test(error.message), JSON.stringify(step));
assert.throws(() => parseReplayPlan({ steps: [] }), /no steps/);
assert.throws(() => parseReplayPlan({ steps: [{ capture: "Home" }], step: [] }), /field "step" it does not take/);
refused({ action: "tap" }, /exactly one of label or identifier/);
refused({ action: "swipe" }, /needs a capture title or an action/);
refused({ action: "constructor" }, /needs a capture title or an action/);
refused({ capture: "Home", platforms: ["web"] }, /platforms must list ios and\/or android/);
// A coordinate tap's fields would turn a label tap into a tap whose risk comes from `target`.
refused({ action: "tap", label: "Delete account", frameId: "frame-1", x: 0.2, y: 0.15, target: "Next" }, /^Step 1 \(tap\) has a field "frameId" it does not take; a tap step takes action, label, identifier, platforms, optional\.$/);
refused({ action: "tap", label: "Next", platform: ["ios"] }, /field "platform"/);
refused({ action: "tap", label: "Next", optinal: true }, /field "optinal"/);
refused({ capture: "Home", optional: true }, /\(capture\) has a field "optional"/);
refused({ capture: "Home", action: "tap", label: "Next" }, /\(capture\) has a field "action"/);
refused({ action: "back", label: "Back" }, /\(back\) has a field "label"/);
refused({ action: "scroll", direction: ["up"] }, /scroll needs a direction/);
refused({ action: "scroll", direction: "sideways" }, /scroll needs a direction/);
refused({ action: "tap", label: "x".repeat(257) }, /label must be 1–256 characters without control characters/);
refused({ action: "tap", identifier: "row\nnext" }, /identifier must be 1–256 characters/);
refused({ action: "fill", label: "Name", text: "two\nlines" }, /fill needs text: one line/);
refused({ action: "type", text: "x".repeat(2001) }, /type needs text/);
refused({ action: "type", text: "{{secret:lowercase}}" }, /invalid secret reference/);
refused({ action: "type", text: "{{secret:OPENAI_API_KEY}}" }, /cannot reference provider/);
refused({ action: "wait", milliseconds: 15_001 }, /between 1 and 15000/);
assert.deepEqual(parseReplayPlan({ steps: [{ capture: "Home" }] }).steps, [{ capture: "Home" }]);
assert.deepEqual(parseReplayPlan({ steps: [{ action: "scroll", direction: "down", optional: true, platforms: ["ios"] }] }).steps,
[{ action: "scroll", direction: "down", platforms: ["ios"], optional: true }]);
});
test("a replay plan with no steps for the platform, or more than the budgets hold, is refused before any device starts", () => {
const limits = { maxScreens: 40, maxActions: 150, maxTurns: 400 };
assert.throws(() => assertReplayFits(parseReplayPlan({ steps: [{ capture: "Home", platforms: ["android"] }, { action: "back", platforms: ["android"] }] }), "ios", limits),
error => error.code === "REPLAY_PLAN_EMPTY" && /No step of the replay plan runs on ios/.test(error.message));
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" }, { action: "back" }, { action: "wait", milliseconds: 100 }] });
assert.doesNotThrow(() => assertReplayFits(plan, "ios", limits));
assert.throws(() => assertReplayFits(plan, "ios", { ...limits, maxScreens: 2, maxActions: 1 }),
error => error.code === "REPLAY_PLAN_OVER_BUDGET" && /2 captures, which need a screen budget of at least 3 \(it is 2\) and 2 actions, more than the action budget of 1/.test(error.message)
&& /--max-screens 3 --max-actions 2/.test(error.nextAction));
assert.throws(() => assertReplayFits(plan, "ios", { ...limits, maxTurns: 8 }), /takes 9 tool calls, more than the 8/);
});
test("a replay that runs out of budget names the planned screens it did not reach in the gap", async () => {
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" }, { action: "back" }, { capture: "Later" }] });
const device = replayDevice();
const session = createReplaySession(plan, "ios");
const { result, map } = await harness([], { session, budgetNote: session.budgetNote, ...device.options, maxScreens: 2 });
assert.equal(result.stopReason, "budget-screens");
assert.match(map.gaps.at(-1).message, /^The exploration stopped at its screens budget with \d+ controls unexplored\. Not captured: Settings, Later\.$/);
});
test("a capture of a screen the plan already captured under another title is reported and does not rename it", async () => {
const plan = parseReplayPlan({ steps: [
{ capture: "Home" }, { capture: "Start" }, { capture: "home" }, { action: "tap", label: "Settings" }, { action: "back" },
] });
const device = replayDevice();
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
assert.equal(result.stopReason, "finished");
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Home → Settings"]);
assert.match(result.summary, /Not captured: Start \(looked the same as Home\)\. Also reached, not in the plan: 1 unnamed screen\.$/);
});
test("a planned state that the explorer merges into an earlier screen is neither reported as unplanned nor left in the map", async () => {
// The README's own example: Dark mode changes the tree, and the capture titled Settings folds that new screen into Settings.
const screens = { ...SCREENS, settingsDark: [...SCREENS.settings, { type: "Button", label: "Theme options" }] };
const plan = parseReplayPlan({ steps: [
{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" },
{ action: "tap", label: "Dark mode" }, { capture: "Settings", state: "dark" },
] });
const device = replayDevice(screens);
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
assert.equal(result.stopReason, "finished");
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Settings"]);
const settings = map.screens.find(screen => screen.title === "Settings");
assert.ok(map.variants.some(variant => variant.screenId === settings.id && variant.name === "dark"));
assert.equal(result.summary, "Replayed 5 planned steps on ios. Every planned screen was captured.");
});
test("a long replay summary is kept within finish's limit, the most useful names first", async () => {
const titles = Array.from({ length: 60 }, (_, index) => `Planned screen ${String(index).padStart(2, "0")} ${"y".repeat(60)}`);
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, ...titles.map(title => ({ capture: title }))] });
const device = replayDevice();
const { result } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options, maxTurns: 200 });
assert.equal(result.stopReason, "finished");
assert.ok(result.summary.length <= 4000, `${result.summary.length} characters`);
assert.match(result.summary, /^Replayed 61 planned steps on ios\. Not captured: Planned screen 00 y+ \(looked the same as Home\), .* and \d+ more\.$/);
});
test("a refused finish ends the replay with stop_blocked instead of a turn budget, and a refused stop is an error", async () => {
const plan = parseReplayPlan({ steps: [{ action: "back" }] });
const calls = [];
const request = refuse => ({ signal: new AbortController().signal, callTool: async (name, input) => {
calls.push({ name, input });
const value = name === "get_progress" ? { captured: [] } : refuse.includes(name) ? { ok: false, code: "AGENT_ACTION_INVALID", message: `${name} refused` } : { ok: true };
return { content: [{ type: "text", text: JSON.stringify(value) }], ...(refuse.includes(name) ? { isError: true } : {}) };
} });
await createReplaySession(plan, "ios").run(request(["finish"]));
assert.deepEqual(calls.map(call => call.name), ["act", "get_progress", "finish", "stop_blocked"]);
assert.match(calls.at(-1).input.reason, /^The replay could not finish \(finish refused\)\. Replayed 1 planned steps on ios\./);
await assert.rejects(createReplaySession(plan, "ios").run(request(["finish", "stop_blocked"])), { code: "REPLAY_STOP_REFUSED" });
});
test("a replay plan's {{secret:NAME}} inputs are read before any device starts and typed for the plan", async () => {
const { runBinaryExploration, runSessionExploration } = await tsImport("../../src/binary/run.ts", import.meta.url);
const { createExplorationInputs } = await tsImport("../../src/config/exploration-inputs.ts", import.meta.url);
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, { action: "fill", label: "Password", text: "{{secret:APP_TEST_PASSWORD}}" }] });
const output = await mkdtemp(join(tmpdir(), "explore-replay-"));
for (const run of [runBinaryExploration, runSessionExploration]) {
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: plan, environment: {} }),
error => error.code === "EXPLORATION_SECRET_MISSING" && /APP_TEST_PASSWORD/.test(error.message));
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: plan, environment: {}, provider: "openai", hints: "Look around" }),
error => error.code === "REPLAY_OPTIONS_IGNORED" && /--provider and --hints would do nothing/.test(error.message));
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: { steps: [{ action: "tap", label: "Next", optinal: true }] }, environment: {} }), { code: "REPLAY_PLAN_INVALID" });
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: parseReplayPlan({ steps: [{ action: "open", url: "https://example.com" }] }), environment: {} }),
error => error.code === "REPLAY_PLAN_INVALID" && /^Step 1: https: URLs leave the app/.test(error.message));
}
const device = replayDevice();
const inputs = createExplorationInputs({ setup: replaySetup(plan), environment: { APP_TEST_PASSWORD: "hunter2-test" } });
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), inputs, ...device.options });
assert.equal(result.stopReason, "finished");
assert.deepEqual(device.acted, [{ action: "fill", label: "Password", text: "hunter2-test" }]);
assert.equal(map.actionAttempts[0].action.input.value, "{{secret:APP_TEST_PASSWORD}}");
assert.ok(!JSON.stringify(map).includes("hunter2-test"));
});
+5 -36
View File
@@ -29,25 +29,25 @@ async function simulatorApp(root) {
}
/** simctl and agent-device on a Mac with one iPhone 17 Pro runtime; the app shows a single screen. */
function localRunner(device = { name: "iPhone 17 Pro", identifier: DEVICE_TYPE }, screen = { nodes: [{ type: "StaticText", label: "Welcome" }, { type: "Button", label: "Settings" }], png: PNG }) {
function localRunner() {
const requests = [];
const ok = data => ({ stdout: JSON.stringify({ success: true, data }), stderr: "" });
const runner = request => {
requests.push(request);
const [first, second, third] = request.args;
if (request.command === "xcrun") {
if (second === "list" && third === "devicetypes") return { stdout: JSON.stringify({ devicetypes: [device] }), stderr: "" };
if (second === "list" && third === "devicetypes") return { stdout: JSON.stringify({ devicetypes: [{ name: "iPhone 17 Pro", identifier: DEVICE_TYPE }] }), stderr: "" };
if (second === "list" && third === "runtimes") return { stdout: JSON.stringify({ runtimes: [{ isAvailable: true, platform: "iOS", identifier: "com.apple.CoreSimulator.SimRuntime.iOS-27-0",
version: "27.0", supportedDeviceTypes: [{ identifier: device.identifier }] }] }), stderr: "" };
version: "27.0", supportedDeviceTypes: [{ identifier: DEVICE_TYPE }] }] }), stderr: "" };
if (second === "create") return { stdout: `${UDID}\n`, stderr: "" };
return { stdout: "", stderr: "" };
}
if (first === "--version") return { stdout: `${controllerVersion("agent-device")}\n`, stderr: "" };
if (first === "open") return ok({ session: CONTROLLER_SESSION, appBundleId: "dev.example.app", device: { platform: "ios", target: "mobile", name: "Explore iPhone 17 Pro", id: UDID },
identifiers: { session: CONTROLLER_SESSION, appBundleId: "dev.example.app", deviceName: "Explore iPhone 17 Pro", deviceId: UDID } });
if (first === "snapshot") return ok({ nodes: screen.nodes });
if (first === "snapshot") return ok({ nodes: [{ type: "StaticText", label: "Welcome" }, { type: "Button", label: "Settings" }] });
if (first === "appstate") return ok({ platform: "ios", appBundleId: "dev.example.app", state: "runningForeground", surface: "app" });
if (first === "screenshot") { writeFileSync(request.args[1], screen.png); return ok({}); }
if (first === "screenshot") { writeFileSync(request.args[1], PNG); return ok({}); }
return ok({});
};
return { requests, runner };
@@ -85,34 +85,3 @@ test("explore:binary --simulator local runs on a simulator it creates, with no E
assert.equal(result.session.id, `local-${UDID}`);
assert.deepEqual(await readdir(join(root, ".cache", "expo-explore", "local-captures")), [], "nothing is left for a sweep");
});
test("an opened iPhone Duo's screenshots are stored upright, and the run records the landscape they show", async t => {
const root = await mkdtemp(join(tmpdir(), "explore-binary-duo-"));
t.after(() => rm(root, { recursive: true, force: true }));
const { createRequire } = await import("node:module");
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
// The inner panel on its portrait axes at 1x, as local agent-device returns it: the tree's root is 669×951 points.
const raw = new Png({ width: 669, height: 951 });
for (let offset = 0; offset < raw.data.length; offset += 4) raw.data.set(offset < 669 * 4 ? [255, 0, 0, 255] : [255, 255, 255, 255], offset);
const nodes = [{ type: "Application", label: "Example", rect: { x: 0, y: 0, width: 669, height: 951 } }, { type: "Button", label: "Settings", rect: { x: 20, y: 40, width: 80, height: 44 } }];
const { runner } = localRunner({ name: "iPhone Duo", identifier: "com.apple.CoreSimulator.SimDeviceType.iPhone-Duo" }, { nodes, png: Png.sync.write(raw) });
const session = { provider: "anthropic", model: "test", async preflight() {}, async run(request) {
const observed = JSON.parse((await request.callTool("observe", { includeImage: true }, request.signal)).content[0].text);
await request.callTool("capture", { frameId: observed.frameId, title: "Home" }, request.signal);
await request.callTool("finish", { summary: "One." }, request.signal).catch(() => {});
return { stopReason: "done" };
} };
const result = await runBinaryExploration({ archive: await simulatorApp(root), output: join(root, "out"), simulator: "local", device: "iPhone Duo", provider: "anthropic", runner,
createSession: () => session, environment: { PATH: process.env.PATH, HOME: root, ANTHROPIC_API_KEY: "key" } });
const { readFile } = await import("node:fs/promises");
const map = JSON.parse(await readFile(result.mapPath, "utf8"));
const artifact = map.artifacts.find(item => item.kind === "screenshot");
assert.deepEqual([artifact.width, artifact.height, artifact.turned], [951, 669, { width: 669, height: 951 }]);
const stored = Png.sync.read(await readFile(join(result.outputDirectory, artifact.path)));
assert.deepEqual([stored.width, stored.height], [951, 669]);
// The panel's top row (red) is the right-hand column once turned.
assert.deepEqual([...stored.data.subarray((951 - 1) * 4, 951 * 4)], [255, 0, 0, 255]);
assert.deepEqual([...stored.data.subarray(0, 4)], [255, 255, 255, 255]);
assert.equal(map.run.environment.orientation, "landscape");
assert.equal(result.runtime.orientation, "landscape");
});