mirror of
https://github.com/expo/explore.git
synced 2026-11-15 13:18:16 -10:00
Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c919c42cac | ||
|
|
90d5d52dab | ||
|
|
7fa07d2cc0 | ||
|
|
6ad0d89c1f |
No files matched your search
@@ -16,6 +16,34 @@ Write "No visible change" if there is none.
|
||||
| --- | --- |
|
||||
| | |
|
||||
|
||||
## Code examples
|
||||
|
||||
<!--
|
||||
Required for any change to how Explore is used or what it writes: a CLI command or flag,
|
||||
config.yaml, a workflow template, a file format (a replay or flows file, report JSON), or a
|
||||
public type. Show it in use as a reader would write it: the command with its flags, the config
|
||||
or file it reads, and what it prints or writes, trimmed to the lines that matter. When existing
|
||||
usage changes, show it before and after.
|
||||
Write "No change to usage" if there is none.
|
||||
-->
|
||||
|
||||
```sh
|
||||
|
||||
```
|
||||
|
||||
## Try it
|
||||
|
||||
<!--
|
||||
Optional, but recommended: something a reviewer can open or run to see the change for themselves.
|
||||
- A Viewer change: the Viewer preview comment on this PR links a live build. Name the report,
|
||||
screen and steps to look at.
|
||||
- A capture or report change: a hosted report from a run of this branch (explore:binary,
|
||||
explore:session or capture with --upload), or an app pull request whose review ran this branch.
|
||||
- Otherwise: the commands that run it from a checkout of this branch, for example
|
||||
`npm ci && npm run build`, then `node dist/cli.js <command>`, with any app or file it needs.
|
||||
Write "Nothing to try" if there is nothing a reviewer could run.
|
||||
-->
|
||||
|
||||
## Testing
|
||||
|
||||
<!-- What you ran and what you checked by hand. -->
|
||||
@@ -174,7 +174,7 @@ Pointed to a build, the [quick start](#quick-start) maps every screen of it, wit
|
||||
- A simulator build. A device `.ipa`, an `.aab`, an unsigned APK, or an APK without native libraries for the emulator (x86_64 on EAS Simulator, `arm64-v8a` locally on Apple silicon; a universal APK has both) is rejected before a device starts.
|
||||
- Platform, bundle id, name, and version are read from the build; you pass nothing else.
|
||||
- A run stops after 15 minutes, 40 screens, or 150 actions (`--max-duration-minutes`, `--max-screens`, `--max-actions`).
|
||||
- Buttons that delete, pay, sign out, send, or leave the app are never tapped. `--allow-destructive "Reset filters"` allows one by its exact label.
|
||||
- Buttons that delete, pay, sign out, send, or leave the app are never tapped. Explore judges a control by its label and its identifier (`btn_delete`, `deleteAccount`), including one tapped by position. `--allow-destructive "Reset filters"` allows one by its exact label.
|
||||
- `--hints "…"` tells the model what to focus on or how to sign in ([details](#sign-in-and-other-app-state)).
|
||||
- On EAS Simulator the report is hosted, and the link lasts as long as its EAS Hosting deployment (see [report access](docs/private-review.md#hosted-previews)); `--no-upload` keeps it local. On a local simulator it opens in the local Viewer (`--no-open` or `--json` skips that; `view --dir <folder>` opens it later), and `--upload` hosts it, which needs an EAS login. `--json` prints the result for scripts.
|
||||
- For a link that stays the same from run to run, pass `--alias NAME` and set `EXPLORE_PREVIEW_TOKEN` to a fixed token of 32 characters from a–z and 2–7 (`LC_ALL=C tr -dc a-z2-7 </dev/urandom | head -c 32`).
|
||||
@@ -508,7 +508,7 @@ npx @expo/explore explore:binary ./my-app.app \
|
||||
--hints "Sign in as [email protected] with {{secret:APP_TEST_PASSWORD}}"
|
||||
```
|
||||
|
||||
The model only sees the reference, and Explore types the value. The value is redacted everywhere, and a screen that shows it gets no screenshot, so keep non-sensitive values like a test email as plain text. In CI, store the value as a repository secret instead.
|
||||
The model only sees the reference, and Explore types the value. The value is redacted everywhere, and a screen that shows it gets no screenshot, so keep non-sensitive values like a test email as plain text. Explore finds the value in the screen's accessibility tree; a screen that draws it without exposing it there (in a canvas or an image) is not recognised. In CI, store the value as a repository secret instead.
|
||||
|
||||
If the sign-in can't be typed (one-time codes, email links, Sign in with Apple or Google), sign in by hand on a session and run `explore:session`. Explore never taps Sign out.
|
||||
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@expo/explore",
|
||||
"version": "0.22.0",
|
||||
"version": "0.22.1",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@expo/explore",
|
||||
"version": "0.22.0",
|
||||
"version": "0.22.1",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@expo/repack-app": "0.10.1",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@expo/explore",
|
||||
"version": "0.22.0",
|
||||
"version": "0.22.1",
|
||||
"description": "Source-driven mobile app maps, deterministic CI capture, and visual reviews",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -55,7 +55,7 @@ set -a && . ./.env.explore && set +a && npx @expo/explore explore:binary ./app.a
|
||||
--hints "Sign in with {{secret:APP_TEST_EMAIL}} and {{secret:APP_TEST_PASSWORD}}, then explore the signed-in app"
|
||||
```
|
||||
|
||||
The model only ever sees the reference; Explore types the value into the field. The value is redacted from the map, the report and errors, and a frame that shows it (the filled-in form, or a profile screen that displays the email) is kept without a screenshot. So make only what is sensitive a secret: a dedicated test email can go in the hints as plain text, and then the account screen keeps its picture. A reference to a variable that is not set fails before a session starts (`EXPLORATION_SECRET_MISSING`); provider keys, `EXPO_TOKEN` and CI variables cannot be referenced.
|
||||
The model only ever sees the reference; Explore types the value into the field. The value is redacted from the map, the report and errors, and a frame that shows it (the filled-in form, or a profile screen that displays the email) is kept without a screenshot. Explore finds the value in the accessibility tree, so a value drawn without an accessible label (in a canvas or an image) is not recognised. So make only what is sensitive a secret: a dedicated test email can go in the hints as plain text, and then the account screen keeps its picture. A reference to a variable that is not set fails before a session starts (`EXPLORATION_SECRET_MISSING`); provider keys, `EXPO_TOKEN` and CI variables cannot be referenced.
|
||||
|
||||
**When a sign-in cannot be typed** (a one-time code, an email link, Sign in with Apple or Google, a captcha): sign in by hand on a session and use `explore:session`. Sign out is never tapped, so the run stays signed in.
|
||||
|
||||
@@ -64,7 +64,7 @@ The model only ever sees the reference; Explore types the value into the field.
|
||||
- Onboarding: `--launch-url` to a screen past it, or hints such as "skip the tour with Skip".
|
||||
- Screens that need data: a test account that already has it, and hints naming where it is ("open the first project").
|
||||
- Permission prompts are denied. To explore what is behind one, allow it by hand on a session first, then `explore:session`.
|
||||
- Purchases, deletes, sends and sign out are blocked. `--allow-destructive "Label"` allows one control by label; a paywall needs an account that already has the entitlement.
|
||||
- Purchases, deletes, sends and sign out are blocked, judged by a control's label and its identifier, including a control tapped by position. `--allow-destructive "Label"` allows one control by label; a paywall needs an account that already has the entitlement.
|
||||
- A large app: raise the budgets (`--max-duration-minutes`, `--max-screens`, `--max-actions`; see [What happens](#what-happens)) and use the hints to say what to cover first.
|
||||
|
||||
## Already have a session? `explore:session`
|
||||
|
||||
+30
-11
@@ -69,6 +69,8 @@ export type BlackBoxExplorerResult = {
|
||||
const DESTRUCTIVE = /\b(delete|remove|erase|reset|wipe|clear all|sign ?out|log ?out|unsubscribe|cancel subscription|deactivate|close account|purchase|buy|subscribe|pay|checkout|send|post|publish|report|block|submit order|confirm order|place order)\b/i;
|
||||
// A bare "Map" is usually the app's own map tab or screen; only a named maps app or directions leave the app.
|
||||
const EXTERNAL = /\b(open in|safari|chrome|browser|app store|play store|share|mail|call|sms|(apple|google) maps|directions)\b/i;
|
||||
/** For the person running Explore, who sees blocked steps in flow and replay results; the model cannot pass it. */
|
||||
const ALLOW_HINT = "(Whoever runs Explore can allow a safe control with --allow-destructive and its exact label.)";
|
||||
const VOLATILE = /\d|[:%$€£]|\b(am|pm)\b/gi;
|
||||
const INTERACTIVE = new Set(["button", "link", "cell", "switch", "tab", "menuitem", "textfield", "securetextfield", "searchfield", "textview", "edittext", "slider", "checkbox", "radiobutton", "segmentedcontrol", "toggle"]);
|
||||
|
||||
@@ -185,13 +187,20 @@ function isInteractive(node: Record<string, unknown>): boolean {
|
||||
return traits.some(trait => /button|link|tab|adjustable/i.test(trait));
|
||||
}
|
||||
|
||||
const allowed = (label: string, allow: readonly string[]) => allow.some(item => item.toLowerCase() === label.toLowerCase());
|
||||
|
||||
export function riskOf(label: string, allow: readonly string[]): ActionRisk {
|
||||
if (allow.some(allowed => allowed.toLowerCase() === label.toLowerCase())) return "safe";
|
||||
if (DESTRUCTIVE.test(label)) return "destructive";
|
||||
if (EXTERNAL.test(label)) return "external";
|
||||
if (allowed(label, allow)) return "safe";
|
||||
// Identifiers join their words (btn_delete, deleteAccount, sign-out), which \b does not split.
|
||||
const words = label.replace(/(\p{Ll}|\d)(\p{Lu})/gu, "$1 $2").replace(/[_.-]+/g, " ");
|
||||
if (DESTRUCTIVE.test(words)) return "destructive";
|
||||
if (EXTERNAL.test(words)) return "external";
|
||||
return "safe";
|
||||
}
|
||||
|
||||
const RISK_ORDER: readonly ActionRisk[] = ["safe", "unknown", "external", "destructive"];
|
||||
const worstRisk = (...risks: ActionRisk[]): ActionRisk => risks.reduce((worst, risk) => RISK_ORDER.indexOf(risk) > RISK_ORDER.indexOf(worst) ? risk : worst, "safe");
|
||||
|
||||
/** Structure plus stable text, with values and numerals masked, so a clock tick or a counter does not create a new screen. */
|
||||
/**
|
||||
* A text field's label is whatever it shows: its placeholder while empty (Android sets `hintShowing`), then the typed text.
|
||||
@@ -660,7 +669,11 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
if (input.frameId !== undefined) {
|
||||
const point = { action: "tap" as const, frameId: text(input.frameId, "frameId", 256), x: Number(input.x), y: Number(input.y) };
|
||||
const target = input.target !== undefined ? cleanLabel(text(input.target, "target", 80)) : "";
|
||||
return { action: point, kind: "press", label: target || placeUnder(point) || `tap ${point.x.toFixed(2)},${point.y.toFixed(2)}`, risk: target ? riskOf(target, allow) : "unknown" };
|
||||
// The control under the point is judged by its own label and identifier too, not only by the model's description of it.
|
||||
const under = nodeUnder(point), underLabel = str(under?.label);
|
||||
const risk = underLabel && allowed(underLabel, allow) ? "safe"
|
||||
: worstRisk(target ? riskOf(target, allow) : "unknown", ...[underLabel, str(under?.identifier)].flatMap(name => name ? [riskOf(name, allow)] : []));
|
||||
return { action: point, kind: "press", label: target || placeUnder(point) || `tap ${point.x.toFixed(2)},${point.y.toFixed(2)}`, risk };
|
||||
}
|
||||
if (!selector) throw failure("AGENT_ACTION_INVALID", "tap needs a label, an identifier, or frame coordinates.");
|
||||
return { action: { action: "tap", ...selector } as BlackBoxAction, kind: "press", label, risk };
|
||||
@@ -714,13 +727,12 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
return { action: "tap", frameId: latest.frameId ?? "current", x: (bounds.x + bounds.width / 2) / extent.width, y: (bounds.y + bounds.height / 2) / extent.height };
|
||||
};
|
||||
|
||||
/** A coordinate tap the model did not describe, on a node the map has no name for (an icon-font button in a header), is
|
||||
* named by what it is and where it sits: "icon button, top right" says more than the coordinates did. */
|
||||
const placeUnder = (point: { x: number; y: number }): string | undefined => {
|
||||
/** The smallest hittable control at a frame point in the latest tree, whether or not the map has recorded it. */
|
||||
const nodeUnder = (point: { x: number; y: number }): Record<string, unknown> | undefined => {
|
||||
const extent = latest ? frameExtent(latest.nodes) : undefined;
|
||||
if (!extent) return undefined;
|
||||
const x = point.x * extent.width, y = point.y * extent.height;
|
||||
const under = latest!.nodes
|
||||
return latest!.nodes
|
||||
.filter(node => {
|
||||
const rect = node.rect as { x: number; y: number; width: number; height: number } | undefined;
|
||||
return !!rect && node.hittable === true && x >= rect.x && x <= rect.x + rect.width && y >= rect.y && y <= rect.y + rect.height
|
||||
@@ -730,6 +742,12 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
const ra = a.rect as { width: number; height: number }, rb = b.rect as { width: number; height: number };
|
||||
return ra.width * ra.height - rb.width * rb.height;
|
||||
})[0];
|
||||
};
|
||||
|
||||
/** A coordinate tap the model did not describe, on a node the map has no name for (an icon-font button in a header), is
|
||||
* named by what it is and where it sits: "icon button, top right" says more than the coordinates did. */
|
||||
const placeUnder = (point: { x: number; y: number }): string | undefined => {
|
||||
const under = nodeUnder(point);
|
||||
if (!under) return undefined;
|
||||
const role = roleOf(under);
|
||||
const label = str(under.label) ?? "";
|
||||
@@ -805,7 +823,8 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
: elementUnder(chosen.action, fromScreenId);
|
||||
const { action, kind } = chosen;
|
||||
const label = named ? named.label : chosen.label;
|
||||
const risk = named && chosen.risk !== "destructive" && chosen.risk !== "external" ? riskOf(named.label, allow) : chosen.risk;
|
||||
// A control allowed by its label is allowed however it is targeted, even when its identifier reads as risky.
|
||||
const risk = named && allowed(named.label, allow) ? "safe" : named && chosen.risk !== "destructive" && chosen.risk !== "external" ? riskOf(named.label, allow) : chosen.risk;
|
||||
const startedAt = timestamp();
|
||||
const attemptId = `att_${randomUUID().slice(0, 12)}`;
|
||||
const matched = map.frontier.find(item => item.status === "pending" && item.screenId === fromScreenId && item.actionKind === kind
|
||||
@@ -817,8 +836,8 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
startedAt, completedAt: timestamp(), reason });
|
||||
await persist();
|
||||
return toolResult({ ok: false, blocked: true, reason: risk === "destructive"
|
||||
? `"${label}" looks destructive (delete/reset/sign out/purchase/send). Explore does not activate such controls. Choose another control.`
|
||||
: `"${label}" looks like it opens another app (browser/store/share/mail/maps). Explore does not activate such controls. Choose another control.` });
|
||||
? `"${label}" looks destructive (delete/reset/sign out/purchase/send). Explore does not activate such controls. Choose another control. ${ALLOW_HINT}`
|
||||
: `"${label}" looks like it opens another app (browser/store/share/mail/maps). Explore does not activate such controls. Choose another control. ${ALLOW_HINT}` });
|
||||
}
|
||||
// The map keeps the reference the model typed; only the device gets the value.
|
||||
const typed = action.action === "fill" || action.action === "type" ? { ...action, text: inputs.resolve(action.text) } : action;
|
||||
|
||||
@@ -1207,3 +1207,52 @@ test("a replay plan's {{secret:NAME}} inputs are read before any device starts a
|
||||
assert.equal(map.actionAttempts[0].action.input.value, "{{secret:APP_TEST_PASSWORD}}");
|
||||
assert.ok(!JSON.stringify(map).includes("hunter2-test"));
|
||||
});
|
||||
|
||||
test("risk words are found inside identifiers that join them", async () => {
|
||||
const { riskOf } = await tsImport("../../src/binary/explorer.ts", import.meta.url);
|
||||
for (const label of ["btn_delete", "deleteAccount", "sign-out", "logOutButton", "settings.reset"]) assert.equal(riskOf(label, []), "destructive", label);
|
||||
assert.equal(riskOf("openInSafari", []), "external");
|
||||
for (const label of ["postsTab", "settings_button", "Profile", "MapView"]) assert.equal(riskOf(label, []), "safe", label);
|
||||
assert.equal(riskOf("resetFilters", ["resetFilters"]), "safe", "an allowed identifier");
|
||||
});
|
||||
|
||||
/**
|
||||
* A screen whose only control is a hittable custom view the map does not record as a control (type Other, as many
|
||||
* custom pressables are): its risk is in its own label and identifier.
|
||||
*/
|
||||
function iconScreen(icon) {
|
||||
return [{ type: "Window", rect: { x: 0, y: 0, width: 400, height: 800 } }, { type: "StaticText", label: "Inbox" },
|
||||
{ type: "Other", hittable: true, rect: { x: 340, y: 40, width: 44, height: 44 }, ...icon }];
|
||||
}
|
||||
|
||||
test("a coordinate tap is judged by the control under the point, whatever the model calls it", async () => {
|
||||
for (const icon of [{ identifier: "btn_delete" }, { label: "Delete conversation" }]) {
|
||||
const { calls, acted, map } = await harness([["observe", { includeImage: true }], ["act", { action: "tap", frameId: "frame-0", x: 0.9, y: 0.075, target: "Profile" }], ["finish", { summary: "x" }]],
|
||||
{ observe: async includeImage => ({ nodes: iconScreen(icon), capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png: PNG, imageSha256: "d".repeat(64) } : {}) }) });
|
||||
assert.equal(acted.length, 0, JSON.stringify(icon));
|
||||
assert.equal(calls[1].result.blocked, true);
|
||||
assert.match(calls[1].result.reason, /looks destructive.*--allow-destructive/);
|
||||
assert.equal(map.actionAttempts.at(-1).risk, "destructive");
|
||||
}
|
||||
});
|
||||
|
||||
test("a coordinate tap on a control allowed by its label goes through, even when its identifier reads as risky", async () => {
|
||||
const { acted } = await harness([["observe", { includeImage: true }], ["act", { action: "tap", frameId: "frame-0", x: 0.9, y: 0.075 }], ["finish", { summary: "x" }]],
|
||||
{ allowDestructive: ["Reset filters"],
|
||||
observe: async includeImage => ({ nodes: iconScreen({ label: "Reset filters", identifier: "resetFilters" }), capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png: PNG, imageSha256: "d".repeat(64) } : {}) }) });
|
||||
assert.equal(acted.length, 1);
|
||||
});
|
||||
|
||||
test("a coordinate tap with no control under the point is still recorded as unknown, not refused", async () => {
|
||||
const { acted, map } = await harness([["observe", { includeImage: true }], ["act", { action: "tap", frameId: "frame-0", x: 0.5, y: 0.5 }], ["finish", { summary: "x" }]],
|
||||
{ observe: async includeImage => ({ nodes: iconScreen({ identifier: "btn_delete" }), capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png: PNG, imageSha256: "d".repeat(64) } : {}) }) });
|
||||
assert.equal(acted.length, 1);
|
||||
assert.equal(map.actionAttempts.at(-1).risk, "unknown");
|
||||
});
|
||||
|
||||
test("a tap by identifier on a control allowed by its label goes through", async () => {
|
||||
const screen = [{ type: "StaticText", label: "Filters" }, { type: "Button", label: "Reset filters", identifier: "resetFilters" }];
|
||||
const { acted } = await harness([["observe", { includeImage: true }], ["act", { action: "tap", identifier: "resetFilters" }], ["finish", { summary: "x" }]],
|
||||
{ allowDestructive: ["Reset filters"], observe: async includeImage => ({ nodes: screen, capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png: PNG, imageSha256: "e".repeat(64) } : {}) }) });
|
||||
assert.equal(acted.length, 1);
|
||||
});
|
||||
@@ -316,9 +316,13 @@
|
||||
.compare-stage.mode-flip .compare-frame-wrap,
|
||||
.sidebar-review-stage.mode-flip .compare-frame-wrap,
|
||||
.sidebar-review-stage.mode-split.landscape .compare-frame-wrap { display: flex; justify-content: center; }
|
||||
/* The width is spelled out, not left to aspect-ratio: WebKit sizes a flex item by its height + aspect-ratio box's
|
||||
available width rather than its ratio width, so Split's two sides stayed half the stage each and each frame sat at
|
||||
the left of its side, off centre under its name (seen on a folded iPhone Duo). */
|
||||
.compare-frame {
|
||||
position: relative;
|
||||
display: block;
|
||||
width: calc(var(--compare-frame-height) * var(--compare-aspect));
|
||||
height: var(--compare-frame-height);
|
||||
aspect-ratio: var(--compare-aspect);
|
||||
overflow: hidden;
|
||||
|
||||
Reference in new issue
Block a user