Compare commits

..
Author SHA1 Message Date
Brent Vatne feed89a477 Release 0.22.1 2026-10-23 17:56:40 -07:00
6 changed files with 15 additions and 115 deletions

No files matched your search

-28
View File
@@ -16,34 +16,6 @@ Write "No visible change" if there is none.
| --- | --- |
| | |
## Code examples
<!--
Required for any change to how Explore is used or what it writes: a CLI command or flag,
config.yaml, a workflow template, a file format (a replay or flows file, report JSON), or a
public type. Show it in use as a reader would write it: the command with its flags, the config
or file it reads, and what it prints or writes, trimmed to the lines that matter. When existing
usage changes, show it before and after.
Write "No change to usage" if there is none.
-->
```sh
```
## Try it
<!--
Optional, but recommended: something a reviewer can open or run to see the change for themselves.
- A Viewer change: the Viewer preview comment on this PR links a live build. Name the report,
screen and steps to look at.
- A capture or report change: a hosted report from a run of this branch (explore:binary,
explore:session or capture with --upload), or an app pull request whose review ran this branch.
- Otherwise: the commands that run it from a checkout of this branch, for example
`npm ci && npm run build`, then `node dist/cli.js <command>`, with any app or file it needs.
Write "Nothing to try" if there is nothing a reviewer could run.
-->
## Testing
<!-- What you ran and what you checked by hand. -->
+2 -2
View File
@@ -174,7 +174,7 @@ Pointed to a build, the [quick start](#quick-start) maps every screen of it, wit
- A simulator build. A device `.ipa`, an `.aab`, an unsigned APK, or an APK without native libraries for the emulator (x86_64 on EAS Simulator, `arm64-v8a` locally on Apple silicon; a universal APK has both) is rejected before a device starts.
- Platform, bundle id, name, and version are read from the build; you pass nothing else.
- A run stops after 15 minutes, 40 screens, or 150 actions (`--max-duration-minutes`, `--max-screens`, `--max-actions`).
- Buttons that delete, pay, sign out, send, or leave the app are never tapped. Explore judges a control by its label and its identifier (`btn_delete`, `deleteAccount`), including one tapped by position. `--allow-destructive "Reset filters"` allows one by its exact label.
- Buttons that delete, pay, sign out, send, or leave the app are never tapped. `--allow-destructive "Reset filters"` allows one by its exact label.
- `--hints "…"` tells the model what to focus on or how to sign in ([details](#sign-in-and-other-app-state)).
- On EAS Simulator the report is hosted, and the link lasts as long as its EAS Hosting deployment (see [report access](docs/private-review.md#hosted-previews)); `--no-upload` keeps it local. On a local simulator it opens in the local Viewer (`--no-open` or `--json` skips that; `view --dir <folder>` opens it later), and `--upload` hosts it, which needs an EAS login. `--json` prints the result for scripts.
- For a link that stays the same from run to run, pass `--alias NAME` and set `EXPLORE_PREVIEW_TOKEN` to a fixed token of 32 characters from a–z and 2–7 (`LC_ALL=C tr -dc a-z2-7 </dev/urandom | head -c 32`).
@@ -508,7 +508,7 @@ npx @expo/explore explore:binary ./my-app.app \
--hints "Sign in as [email protected] with {{secret:APP_TEST_PASSWORD}}"
```
The model only sees the reference, and Explore types the value. The value is redacted everywhere, and a screen that shows it gets no screenshot, so keep non-sensitive values like a test email as plain text. Explore finds the value in the screen's accessibility tree; a screen that draws it without exposing it there (in a canvas or an image) is not recognised. In CI, store the value as a repository secret instead.
The model only sees the reference, and Explore types the value. The value is redacted everywhere, and a screen that shows it gets no screenshot, so keep non-sensitive values like a test email as plain text. In CI, store the value as a repository secret instead.
If the sign-in can't be typed (one-time codes, email links, Sign in with Apple or Google), sign in by hand on a session and run `explore:session`. Explore never taps Sign out.
+2 -2
View File
@@ -55,7 +55,7 @@ set -a && . ./.env.explore && set +a && npx @expo/explore explore:binary ./app.a
--hints "Sign in with {{secret:APP_TEST_EMAIL}} and {{secret:APP_TEST_PASSWORD}}, then explore the signed-in app"
```
The model only ever sees the reference; Explore types the value into the field. The value is redacted from the map, the report and errors, and a frame that shows it (the filled-in form, or a profile screen that displays the email) is kept without a screenshot. Explore finds the value in the accessibility tree, so a value drawn without an accessible label (in a canvas or an image) is not recognised. So make only what is sensitive a secret: a dedicated test email can go in the hints as plain text, and then the account screen keeps its picture. A reference to a variable that is not set fails before a session starts (`EXPLORATION_SECRET_MISSING`); provider keys, `EXPO_TOKEN` and CI variables cannot be referenced.
The model only ever sees the reference; Explore types the value into the field. The value is redacted from the map, the report and errors, and a frame that shows it (the filled-in form, or a profile screen that displays the email) is kept without a screenshot. So make only what is sensitive a secret: a dedicated test email can go in the hints as plain text, and then the account screen keeps its picture. A reference to a variable that is not set fails before a session starts (`EXPLORATION_SECRET_MISSING`); provider keys, `EXPO_TOKEN` and CI variables cannot be referenced.
**When a sign-in cannot be typed** (a one-time code, an email link, Sign in with Apple or Google, a captcha): sign in by hand on a session and use `explore:session`. Sign out is never tapped, so the run stays signed in.
@@ -64,7 +64,7 @@ The model only ever sees the reference; Explore types the value into the field.
- Onboarding: `--launch-url` to a screen past it, or hints such as "skip the tour with Skip".
- Screens that need data: a test account that already has it, and hints naming where it is ("open the first project").
- Permission prompts are denied. To explore what is behind one, allow it by hand on a session first, then `explore:session`.
- Purchases, deletes, sends and sign out are blocked, judged by a control's label and its identifier, including a control tapped by position. `--allow-destructive "Label"` allows one control by label; a paywall needs an account that already has the entitlement.
- Purchases, deletes, sends and sign out are blocked. `--allow-destructive "Label"` allows one control by label; a paywall needs an account that already has the entitlement.
- A large app: raise the budgets (`--max-duration-minutes`, `--max-screens`, `--max-actions`; see [What happens](#what-happens)) and use the hints to say what to cover first.
## Already have a session? `explore:session`
+11 -30
View File
@@ -69,8 +69,6 @@ export type BlackBoxExplorerResult = {
const DESTRUCTIVE = /\b(delete|remove|erase|reset|wipe|clear all|sign ?out|log ?out|unsubscribe|cancel subscription|deactivate|close account|purchase|buy|subscribe|pay|checkout|send|post|publish|report|block|submit order|confirm order|place order)\b/i;
// A bare "Map" is usually the app's own map tab or screen; only a named maps app or directions leave the app.
const EXTERNAL = /\b(open in|safari|chrome|browser|app store|play store|share|mail|call|sms|(apple|google) maps|directions)\b/i;
/** For the person running Explore, who sees blocked steps in flow and replay results; the model cannot pass it. */
const ALLOW_HINT = "(Whoever runs Explore can allow a safe control with --allow-destructive and its exact label.)";
const VOLATILE = /\d|[:%$€£]|\b(am|pm)\b/gi;
const INTERACTIVE = new Set(["button", "link", "cell", "switch", "tab", "menuitem", "textfield", "securetextfield", "searchfield", "textview", "edittext", "slider", "checkbox", "radiobutton", "segmentedcontrol", "toggle"]);
@@ -187,20 +185,13 @@ function isInteractive(node: Record<string, unknown>): boolean {
return traits.some(trait => /button|link|tab|adjustable/i.test(trait));
}
const allowed = (label: string, allow: readonly string[]) => allow.some(item => item.toLowerCase() === label.toLowerCase());
export function riskOf(label: string, allow: readonly string[]): ActionRisk {
if (allowed(label, allow)) return "safe";
// Identifiers join their words (btn_delete, deleteAccount, sign-out), which \b does not split.
const words = label.replace(/(\p{Ll}|\d)(\p{Lu})/gu, "$1 $2").replace(/[_.-]+/g, " ");
if (DESTRUCTIVE.test(words)) return "destructive";
if (EXTERNAL.test(words)) return "external";
if (allow.some(allowed => allowed.toLowerCase() === label.toLowerCase())) return "safe";
if (DESTRUCTIVE.test(label)) return "destructive";
if (EXTERNAL.test(label)) return "external";
return "safe";
}
const RISK_ORDER: readonly ActionRisk[] = ["safe", "unknown", "external", "destructive"];
const worstRisk = (...risks: ActionRisk[]): ActionRisk => risks.reduce((worst, risk) => RISK_ORDER.indexOf(risk) > RISK_ORDER.indexOf(worst) ? risk : worst, "safe");
/** Structure plus stable text, with values and numerals masked, so a clock tick or a counter does not create a new screen. */
/**
* A text field's label is whatever it shows: its placeholder while empty (Android sets `hintShowing`), then the typed text.
@@ -669,11 +660,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
if (input.frameId !== undefined) {
const point = { action: "tap" as const, frameId: text(input.frameId, "frameId", 256), x: Number(input.x), y: Number(input.y) };
const target = input.target !== undefined ? cleanLabel(text(input.target, "target", 80)) : "";
// The control under the point is judged by its own label and identifier too, not only by the model's description of it.
const under = nodeUnder(point), underLabel = str(under?.label);
const risk = underLabel && allowed(underLabel, allow) ? "safe"
: worstRisk(target ? riskOf(target, allow) : "unknown", ...[underLabel, str(under?.identifier)].flatMap(name => name ? [riskOf(name, allow)] : []));
return { action: point, kind: "press", label: target || placeUnder(point) || `tap ${point.x.toFixed(2)},${point.y.toFixed(2)}`, risk };
return { action: point, kind: "press", label: target || placeUnder(point) || `tap ${point.x.toFixed(2)},${point.y.toFixed(2)}`, risk: target ? riskOf(target, allow) : "unknown" };
}
if (!selector) throw failure("AGENT_ACTION_INVALID", "tap needs a label, an identifier, or frame coordinates.");
return { action: { action: "tap", ...selector } as BlackBoxAction, kind: "press", label, risk };
@@ -727,12 +714,13 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
return { action: "tap", frameId: latest.frameId ?? "current", x: (bounds.x + bounds.width / 2) / extent.width, y: (bounds.y + bounds.height / 2) / extent.height };
};
/** The smallest hittable control at a frame point in the latest tree, whether or not the map has recorded it. */
const nodeUnder = (point: { x: number; y: number }): Record<string, unknown> | undefined => {
/** A coordinate tap the model did not describe, on a node the map has no name for (an icon-font button in a header), is
* named by what it is and where it sits: "icon button, top right" says more than the coordinates did. */
const placeUnder = (point: { x: number; y: number }): string | undefined => {
const extent = latest ? frameExtent(latest.nodes) : undefined;
if (!extent) return undefined;
const x = point.x * extent.width, y = point.y * extent.height;
return latest!.nodes
const under = latest!.nodes
.filter(node => {
const rect = node.rect as { x: number; y: number; width: number; height: number } | undefined;
return !!rect && node.hittable === true && x >= rect.x && x <= rect.x + rect.width && y >= rect.y && y <= rect.y + rect.height
@@ -742,12 +730,6 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
const ra = a.rect as { width: number; height: number }, rb = b.rect as { width: number; height: number };
return ra.width * ra.height - rb.width * rb.height;
})[0];
};
/** A coordinate tap the model did not describe, on a node the map has no name for (an icon-font button in a header), is
* named by what it is and where it sits: "icon button, top right" says more than the coordinates did. */
const placeUnder = (point: { x: number; y: number }): string | undefined => {
const under = nodeUnder(point);
if (!under) return undefined;
const role = roleOf(under);
const label = str(under.label) ?? "";
@@ -823,8 +805,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
: elementUnder(chosen.action, fromScreenId);
const { action, kind } = chosen;
const label = named ? named.label : chosen.label;
// A control allowed by its label is allowed however it is targeted, even when its identifier reads as risky.
const risk = named && allowed(named.label, allow) ? "safe" : named && chosen.risk !== "destructive" && chosen.risk !== "external" ? riskOf(named.label, allow) : chosen.risk;
const risk = named && chosen.risk !== "destructive" && chosen.risk !== "external" ? riskOf(named.label, allow) : chosen.risk;
const startedAt = timestamp();
const attemptId = `att_${randomUUID().slice(0, 12)}`;
const matched = map.frontier.find(item => item.status === "pending" && item.screenId === fromScreenId && item.actionKind === kind
@@ -836,8 +817,8 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
startedAt, completedAt: timestamp(), reason });
await persist();
return toolResult({ ok: false, blocked: true, reason: risk === "destructive"
? `"${label}" looks destructive (delete/reset/sign out/purchase/send). Explore does not activate such controls. Choose another control. ${ALLOW_HINT}`
: `"${label}" looks like it opens another app (browser/store/share/mail/maps). Explore does not activate such controls. Choose another control. ${ALLOW_HINT}` });
? `"${label}" looks destructive (delete/reset/sign out/purchase/send). Explore does not activate such controls. Choose another control.`
: `"${label}" looks like it opens another app (browser/store/share/mail/maps). Explore does not activate such controls. Choose another control.` });
}
// The map keeps the reference the model typed; only the device gets the value.
const typed = action.action === "fill" || action.action === "type" ? { ...action, text: inputs.resolve(action.text) } : action;
-49
View File
@@ -1207,52 +1207,3 @@ test("a replay plan's {{secret:NAME}} inputs are read before any device starts a
assert.equal(map.actionAttempts[0].action.input.value, "{{secret:APP_TEST_PASSWORD}}");
assert.ok(!JSON.stringify(map).includes("hunter2-test"));
});
test("risk words are found inside identifiers that join them", async () => {
const { riskOf } = await tsImport("../../src/binary/explorer.ts", import.meta.url);
for (const label of ["btn_delete", "deleteAccount", "sign-out", "logOutButton", "settings.reset"]) assert.equal(riskOf(label, []), "destructive", label);
assert.equal(riskOf("openInSafari", []), "external");
for (const label of ["postsTab", "settings_button", "Profile", "MapView"]) assert.equal(riskOf(label, []), "safe", label);
assert.equal(riskOf("resetFilters", ["resetFilters"]), "safe", "an allowed identifier");
});
/**
* A screen whose only control is a hittable custom view the map does not record as a control (type Other, as many
* custom pressables are): its risk is in its own label and identifier.
*/
function iconScreen(icon) {
return [{ type: "Window", rect: { x: 0, y: 0, width: 400, height: 800 } }, { type: "StaticText", label: "Inbox" },
{ type: "Other", hittable: true, rect: { x: 340, y: 40, width: 44, height: 44 }, ...icon }];
}
test("a coordinate tap is judged by the control under the point, whatever the model calls it", async () => {
for (const icon of [{ identifier: "btn_delete" }, { label: "Delete conversation" }]) {
const { calls, acted, map } = await harness([["observe", { includeImage: true }], ["act", { action: "tap", frameId: "frame-0", x: 0.9, y: 0.075, target: "Profile" }], ["finish", { summary: "x" }]],
{ observe: async includeImage => ({ nodes: iconScreen(icon), capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png: PNG, imageSha256: "d".repeat(64) } : {}) }) });
assert.equal(acted.length, 0, JSON.stringify(icon));
assert.equal(calls[1].result.blocked, true);
assert.match(calls[1].result.reason, /looks destructive.*--allow-destructive/);
assert.equal(map.actionAttempts.at(-1).risk, "destructive");
}
});
test("a coordinate tap on a control allowed by its label goes through, even when its identifier reads as risky", async () => {
const { acted } = await harness([["observe", { includeImage: true }], ["act", { action: "tap", frameId: "frame-0", x: 0.9, y: 0.075 }], ["finish", { summary: "x" }]],
{ allowDestructive: ["Reset filters"],
observe: async includeImage => ({ nodes: iconScreen({ label: "Reset filters", identifier: "resetFilters" }), capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png: PNG, imageSha256: "d".repeat(64) } : {}) }) });
assert.equal(acted.length, 1);
});
test("a coordinate tap with no control under the point is still recorded as unknown, not refused", async () => {
const { acted, map } = await harness([["observe", { includeImage: true }], ["act", { action: "tap", frameId: "frame-0", x: 0.5, y: 0.5 }], ["finish", { summary: "x" }]],
{ observe: async includeImage => ({ nodes: iconScreen({ identifier: "btn_delete" }), capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png: PNG, imageSha256: "d".repeat(64) } : {}) }) });
assert.equal(acted.length, 1);
assert.equal(map.actionAttempts.at(-1).risk, "unknown");
});
test("a tap by identifier on a control allowed by its label goes through", async () => {
const screen = [{ type: "StaticText", label: "Filters" }, { type: "Button", label: "Reset filters", identifier: "resetFilters" }];
const { acted } = await harness([["observe", { includeImage: true }], ["act", { action: "tap", identifier: "resetFilters" }], ["finish", { summary: "x" }]],
{ allowDestructive: ["Reset filters"], observe: async includeImage => ({ nodes: screen, capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png: PNG, imageSha256: "e".repeat(64) } : {}) }) });
assert.equal(acted.length, 1);
});
@@ -316,13 +316,9 @@
.compare-stage.mode-flip .compare-frame-wrap,
.sidebar-review-stage.mode-flip .compare-frame-wrap,
.sidebar-review-stage.mode-split.landscape .compare-frame-wrap { display: flex; justify-content: center; }
/* The width is spelled out, not left to aspect-ratio: WebKit sizes a flex item by its height + aspect-ratio box's
available width rather than its ratio width, so Split's two sides stayed half the stage each and each frame sat at
the left of its side, off centre under its name (seen on a folded iPhone Duo). */
.compare-frame {
position: relative;
display: block;
width: calc(var(--compare-frame-height) * var(--compare-aspect));
height: var(--compare-frame-height);
aspect-ratio: var(--compare-aspect);
overflow: hidden;