mirror of
https://github.com/expo/explore.git
synced 2026-11-15 17:32:23 -10:00
Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d6b97331b3 | ||
|
|
6394778533 | ||
|
|
66393ddbc9 | ||
|
|
da719b0c31 | ||
|
|
1b0a0a1d96 |
No files matched your search
+14
-3
@@ -29,10 +29,11 @@ This file describes the source layout and the rules that keep it in order.
|
||||
| `review` | Compares base and head evidence, and prepares the local report assets. `ci` calls `evidence-assessment.ts` and `capture-review.ts`. | `evidence-assessment.ts`, `capture-review.ts`, `visual-evidence.ts`, `runtime-evidence.ts`, `evidence-comparator.ts`, `reviewer-changes.ts` |
|
||||
| `report` | The report artifact, the pull-request comment, and the Viewer links. | `artifact.ts`, `pr-comment*.ts`, `viewer-links.ts` |
|
||||
| `ci` | The pipeline behind `source:verify`, `capture:review` and `capture:baseline`. | `pipeline.ts`, `verified-plan.ts`, `artifact-cache.ts` |
|
||||
| `binary` | Explores an app with no source (`explore:binary`, `explore:session`): the archive, the device session, the explorer, and the strategies that drive it. | `run.ts` (entry), `explorer.ts`, `strategy.ts`, `replay.ts`, `flows.ts`, `archive.ts`, `report.ts` |
|
||||
| `cli` | Argument parsing, help, and the command handlers. | `args.ts`, `help.ts`, `*-commands.ts` |
|
||||
| `map` | The local map store, build catalogs and comparisons, and the server for `expo-explore view`. | `store.ts`, `build-workspace.ts`, `compare.ts`, `viewer-server.ts` |
|
||||
| `contracts` | The modules that `viewer/src` imports from `src/`, by value or by type: data shapes, validators and projections that the CLI and the Viewer share. `capture.ts` also defines the review types that the Viewer reads, such as `PublicReview`, and `review-validation.ts` validates them. | `capture.ts`, `review.ts`, `review-validation.ts`, `source-validation.ts` |
|
||||
| `shared` | Small Node utilities that several folders use, and the pinned `agent-device` version that capture and review both check. | `ci-errors.ts`, `canonical-json.ts`, `bounded-bytes.ts`, `png-evidence.ts`, `agent-device-version.ts`, `timings.ts` (a capture job's timings and its summary, CI-FLOW §4.0) |
|
||||
| `shared` | Small Node utilities that several folders use, and the pinned `agent-device` version that capture and review both check. | `ci-errors.ts`, `canonical-json.ts`, `bounded-bytes.ts`, `png-evidence.ts`, `archive-entries.ts` (tar.gz and zip readers), `agent-device-version.ts`, `timings.ts` (a capture job's timings and its summary, CI-FLOW §4.0) |
|
||||
|
||||
## Dependency rule
|
||||
|
||||
@@ -44,14 +45,24 @@ Each layer imports only from the layers below it.
|
||||
3. `source` imports `source/route-providers`.
|
||||
4. `capture` imports `capture/agent`, `config` and `map`. `capture/agent` imports `capture` and `config`.
|
||||
5. `review` imports `capture` and `map`.
|
||||
6. `ci` imports `config`, `source`, `capture`, `capture/agent`, `review` and `report`.
|
||||
7. `cli` imports `ci`, `map`, `config` and `report`. `src/cli.ts` imports `cli`, `map` and `shared`.
|
||||
6. `ci` imports `config`, `source`, `capture`, `capture/agent`, `review` and `report`. `binary` imports `config`, `capture`, `capture/agent`, `map` and `report`.
|
||||
7. `cli` imports `ci`, `binary`, `map`, `config`, `report` and `review`. `src/cli.ts` imports `cli`, `map` and `shared`.
|
||||
|
||||
Every folder may import `contracts/`, `shared/` and `src/types.ts`.
|
||||
The exception is `contracts/`: its value imports stay inside `contracts/` or go to `src/types.ts`.
|
||||
The only cycle is `capture` and `capture/agent`, which are one feature.
|
||||
Do not add a new import that points up a layer.
|
||||
|
||||
## Capture strategies
|
||||
|
||||
A capture decides three things: what to look at, how to get there, and what to record.
|
||||
|
||||
- **What to look at.** For a review, `source` plans the routes from the app's source (`capture-plan.ts`). An exploration has no source; its strategy decides as it goes.
|
||||
- **How to get there.** In a review, `capture/route-capture.ts` opens each planned route by deep link, or, in agent mode, a model navigates to it (`capture/agent/runtime.ts`). In an exploration, a strategy drives the explorer's tools: a model exploring on its own, a replay plan (`binary/replay.ts`), or named flows (`binary/flows.ts`).
|
||||
- **What to record.** The host records the evidence, never the strategy: `capture/route-capture.ts` in a review, `binary/explorer.ts` in an exploration. The host also owns the device, the budgets, refusing destructive and app-leaving actions, and redacting secrets. Both write an `ExploreMap` (`src/types.ts`, `schema/screen-map.v1.schema.json`). A review adds the capture bundle (`contracts/capture.ts`); an exploration adds the report in `contracts/standalone-report.ts`.
|
||||
|
||||
To add an exploration strategy, add a case to `StrategyChoice` in `binary/strategy.ts`: what it checks before any device starts, and the session it runs. The session is given the explorer's tools and calls them, as the model sessions, `createReplaySession` and `createFlowSession` do. Reviews have no such seam yet: the deep-link loop and agent mode are two branches of `capturePlannedRoutes`.
|
||||
|
||||
## The contracts boundary
|
||||
|
||||
`viewer/src` imports `src/` only through `src/contracts/` and `src/types.ts`.
|
||||
|
||||
@@ -267,6 +267,14 @@ npx @expo/explore explore:binary ./my-app.app --flows flows.json
|
||||
- The report lists every flow and checkpoint of the file, in order, even those a run never reached: each one without a picture says why, so two runs still pair up step by step.
|
||||
- Each flow is also a journey of the map: the path its steps took, from the screen it started on. When the app moved without a tap the map could record (it left the app, or changed on its own), the path goes on as a second journey of the flow.
|
||||
- Relaunching ends the app's process, and the app keeps its data, so a flow that signs in leaves the next flow signed in. On `explore:session`, the app is relaunched before the first flow too.
|
||||
|
||||
To compare two runs of the same flows file, such as two builds, pass both output directories:
|
||||
|
||||
```sh
|
||||
npx @expo/explore compare:flows --before explore-app-1 --after explore-app-2 --output flows.json
|
||||
```
|
||||
|
||||
Each checkpoint comes back `unchanged`, `changed` (with where the screenshots differ, the status bar and home indicator set aside, as in a review), `missing` (no picture on one side, with why), `different-size` (another device or orientation, so not compared), or `added` / `removed` (in only one run's flows file). The result also lists how the two runs were captured differently (the app, device, OS version, locale, orientation, or appearance), since a change can come from those.
|
||||
- A `{{secret:NAME}}` or a described step limited by `platforms` to another platform needs no value, and no model, on this one.
|
||||
|
||||
</details>
|
||||
|
||||
@@ -42,6 +42,7 @@ To compare runs, or an iPhone with an Android phone, screen for screen, pass `--
|
||||
When the user describes a flow, or a set of flows, to look at ("sign in, then rename a pattern"), write them as a flows file and pass `--flows flows.json`, not `--replay`. The format is in the README ("Follow named flows, step by step"): `{ "flows": [{ "name": "Sign in", "steps": [{ "checkpoint": "Welcome" }, { "do": "Sign in with {{secret:TEST_EMAIL}}" }, { "checkpoint": "Home" }] }] }`.
|
||||
- Each step is an exact action (the `--replay` format), a step described in words (`do`) that a model carries out, or a named `checkpoint`. Use exact actions where you know the labels, and `do` where you don't.
|
||||
- Each flow starts from a relaunched app, and its checkpoints are photographed under their names. `report/exploration.json` lists each flow under `exploration.flows`, with its status and its checkpoints in order, so two runs pair up by flow and checkpoint name. A checkpoint on a screen that withholds its picture (a secret on it) is listed with why it has none, and its flow goes on.
|
||||
- To compare two runs of the same flows file, run `npx @expo/explore compare:flows --before DIR --after DIR` with the two output directories. Each checkpoint is `unchanged`, `changed` (with regions), `missing`, `different-size`, `added` or `removed`, and `conditions` lists what differed between the runs (device, OS, locale, appearance).
|
||||
- The file is checked before a device starts (`FLOWS_PLAN_INVALID`, `FLOWS_PLAN_EMPTY`, `FLOWS_PLAN_OVER_BUDGET`). `--provider`, `--model` and `--hints` steer the model that carries out `do` steps, and are refused when there are none (`FLOWS_OPTIONS_IGNORED`). `--flows` and `--replay` cannot be combined.
|
||||
|
||||
## Sign-in and other app state
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import { CiError } from "../shared/ci-errors.js";
|
||||
|
||||
/** Schemes the OS routes to a browser, the store, Settings, or another app: opening one hands the session away from the app under test. */
|
||||
const SYSTEM_URL_SCHEMES = /^(https?|ftp|file|javascript|data|blob|mailto|tel|telprompt|facetime(-audio)?|sms|mms|imessage|itms(-apps|-services|-beta)?|macappstore|app-settings|prefs|shortcuts|maps|music|videos|photos-redirect|x-apple-[a-z-]+|market|intent|content|android-app|geo|package):$/i;
|
||||
|
||||
const urlError = (code: string, message: string, nextAction: string) => new CiError({ code, stage: "session", message, nextAction });
|
||||
|
||||
/** Only the app's own URLs may be opened; an https page or a store link would hand the session to another app. */
|
||||
export function assertAppUrl(url: string, appSchemes: readonly string[] = []): void {
|
||||
let parsed: URL;
|
||||
try { parsed = new URL(url); } catch { throw urlError("AGENT_URL_INVALID", `Not a URL: ${url}`, "Pass a custom-scheme app URL."); }
|
||||
if (SYSTEM_URL_SCHEMES.test(parsed.protocol)) throw urlError("AGENT_URL_NOT_ALLOWED", `${parsed.protocol} URLs leave the app; only the app's own URL schemes are allowed.`, "Use the app's own URL scheme.");
|
||||
const scheme = parsed.protocol.slice(0, -1).toLowerCase();
|
||||
if (appSchemes.length && !appSchemes.includes(scheme)) {
|
||||
throw urlError("AGENT_URL_NOT_ALLOWED", `${parsed.protocol} is not a URL scheme this app registers (${appSchemes.join(", ")}).`, "Use one of the app's own URL schemes.");
|
||||
}
|
||||
}
|
||||
+1
-126
@@ -2,9 +2,7 @@ import { createHash } from "node:crypto";
|
||||
import { createReadStream } from "node:fs";
|
||||
import { lstat, mkdir, open, readdir, readFile, realpath, writeFile, chmod } from "node:fs/promises";
|
||||
import { basename, join, relative, resolve, sep } from "node:path";
|
||||
import { createGunzip } from "node:zlib";
|
||||
import { Reader, Uint8ArrayWriter, ZipReader, configure } from "@zip.js/zip.js";
|
||||
import { CiError } from "../shared/ci-errors.js";
|
||||
import { MAX_ARCHIVE_ENTRIES, MAX_METADATA_BYTES, archiveError, readZipEntry, safeRelativePath, walkTar, zipEntries } from "../shared/archive-entries.js";
|
||||
import { APP_ICON_SOURCE_LIMIT, isWebp, shrinkAppIcon, webpSize } from "../capture/app-icon.js";
|
||||
import { resolveResourceStrings } from "./android-resources.js";
|
||||
import type { Platform } from "../types.js";
|
||||
@@ -38,14 +36,8 @@ export type InspectedArchive = {
|
||||
};
|
||||
|
||||
export const DEFAULT_MAX_ARCHIVE_BYTES = 1024 * 1024 * 1024;
|
||||
const MAX_METADATA_BYTES = 4 * 1024 * 1024;
|
||||
const MAX_ARCHIVE_ENTRIES = 200_000;
|
||||
const MAX_RESOURCE_TABLE_BYTES = 32 * 1024 * 1024;
|
||||
|
||||
export function archiveError(code: string, message: string, nextAction: string, details?: Record<string, unknown>): CiError {
|
||||
return new CiError({ code, stage: "archive", message, nextAction, ...(details ? { details } : {}) });
|
||||
}
|
||||
|
||||
/** The archive kind from its name; the content check follows and may still reject it. */
|
||||
export function archiveKind(path: string): ArchiveKind {
|
||||
const name = basename(path).toLowerCase();
|
||||
@@ -231,63 +223,6 @@ async function hashDirectory(root: string, maxBytes: number): Promise<{ sha256:
|
||||
return { sha256: hash.digest("hex"), bytes };
|
||||
}
|
||||
|
||||
// ---- tar.gz -------------------------------------------------------------------------------------
|
||||
|
||||
type TarVisitor = (entry: { name: string; type: string; size: number; mode: number }, contents: () => Promise<Buffer>, skip: () => Promise<void>) => Promise<boolean | void>;
|
||||
|
||||
/** Walk a gzip tarball once; `visit` returns true to stop. Long GNU names are honoured; nothing is written. */
|
||||
export async function walkTar(path: string, visit: TarVisitor): Promise<void> {
|
||||
const stream = createReadStream(path).pipe(createGunzip());
|
||||
const iterator = stream[Symbol.asyncIterator]();
|
||||
let buffer = Buffer.alloc(0);
|
||||
let done = false;
|
||||
const fill = async (needed: number) => {
|
||||
while (buffer.length < needed && !done) {
|
||||
const next = await iterator.next();
|
||||
if (next.done) { done = true; break; }
|
||||
buffer = Buffer.concat([buffer, next.value as Buffer]);
|
||||
}
|
||||
};
|
||||
const take = async (count: number): Promise<Buffer> => {
|
||||
await fill(count);
|
||||
if (buffer.length < count) throw archiveError("ARCHIVE_INVALID", "The tarball ended unexpectedly.", "Pass a complete .tar.gz archive.");
|
||||
const out = buffer.subarray(0, count); buffer = buffer.subarray(count); return Buffer.from(out);
|
||||
};
|
||||
const drop = async (count: number) => {
|
||||
let remaining = count;
|
||||
while (remaining > 0) {
|
||||
await fill(Math.min(remaining, 1 << 20));
|
||||
const step = Math.min(remaining, buffer.length);
|
||||
if (!step) throw archiveError("ARCHIVE_INVALID", "The tarball ended unexpectedly.", "Pass a complete .tar.gz archive.");
|
||||
buffer = buffer.subarray(step); remaining -= step;
|
||||
}
|
||||
};
|
||||
let longName: string | undefined;
|
||||
let entries = 0;
|
||||
try {
|
||||
for (;;) {
|
||||
await fill(512);
|
||||
if (buffer.length < 512 || buffer.subarray(0, 512).every(byte => byte === 0)) return;
|
||||
const header = await take(512);
|
||||
const field = (offset: number, length: number) => header.subarray(offset, offset + length).toString("utf8").replace(/\0.*$/s, "");
|
||||
const size = parseInt(field(124, 12).trim() || "0", 8);
|
||||
const type = field(156, 1) || "0";
|
||||
const prefix = field(345, 155);
|
||||
const name = longName ?? (prefix ? `${prefix}/${field(0, 100)}` : field(0, 100));
|
||||
longName = undefined;
|
||||
const padded = Math.ceil(size / 512) * 512;
|
||||
if (type === "L") { longName = (await take(padded)).subarray(0, size).toString("utf8").replace(/\0+$/, ""); continue; }
|
||||
if (++entries > MAX_ARCHIVE_ENTRIES) throw archiveError("ARCHIVE_TOO_LARGE", "The tarball has too many entries.", "Pass a smaller archive.");
|
||||
let consumed = false;
|
||||
const stop = await visit({ name, type, size, mode: parseInt(field(100, 8).trim() || "0", 8) },
|
||||
async () => { consumed = true; return (await take(padded)).subarray(0, size); },
|
||||
async () => { consumed = true; await drop(padded); });
|
||||
if (!consumed) await drop(padded);
|
||||
if (stop) return;
|
||||
}
|
||||
} finally { await iterator.return?.(undefined); }
|
||||
}
|
||||
|
||||
async function tarEntry(path: string, match: (name: string) => boolean): Promise<{ name: string; contents: Buffer } | undefined> {
|
||||
let found: { name: string; contents: Buffer } | undefined;
|
||||
await walkTar(path, async (entry, contents) => {
|
||||
@@ -299,12 +234,6 @@ async function tarEntry(path: string, match: (name: string) => boolean): Promise
|
||||
return found;
|
||||
}
|
||||
|
||||
export function safeRelativePath(name: string): string | undefined {
|
||||
const parts = name.replace(/^\.\//, "").split("/").filter(part => part.length);
|
||||
if (!parts.length || parts.some(part => part === "." || part === ".." || part.includes("\\") || part.includes("\0"))) return undefined;
|
||||
return parts.join("/");
|
||||
}
|
||||
|
||||
/**
|
||||
* Unpack the iOS bundle out of a .tar.gz or simulator .ipa into `destination` for installation.
|
||||
* Paths are confined to the destination; symbolic and hard links are rejected (an app bundle has none).
|
||||
@@ -347,60 +276,6 @@ export async function extractIosBundle(archive: InspectedArchive, destination: s
|
||||
return appDirectory;
|
||||
}
|
||||
|
||||
// ---- zip ----------------------------------------------------------------------------------------
|
||||
|
||||
type ZipEntry = Awaited<ReturnType<ZipReader<unknown>["getEntries"]>>[number];
|
||||
|
||||
/** zip.js reads the central directory and entries by range, so a 200 MB APK is never loaded whole. */
|
||||
class FileRangeReader extends Reader<unknown> {
|
||||
constructor(private readonly path: string) { super(path); }
|
||||
override async init(): Promise<void> {
|
||||
this.size = (await lstat(this.path)).size;
|
||||
}
|
||||
/** Each range opens its own descriptor: nothing stays open after the entries are read, so no handle leaks when a caller stops early. */
|
||||
override async readUint8Array(index: number, length: number): Promise<Uint8Array> {
|
||||
const out = new Uint8Array(length);
|
||||
const handle = await open(this.path, "r");
|
||||
try {
|
||||
let offset = 0;
|
||||
while (offset < length) {
|
||||
const { bytesRead } = await handle.read(out, offset, length - offset, index + offset);
|
||||
if (bytesRead === 0) break;
|
||||
offset += bytesRead;
|
||||
}
|
||||
return offset === length ? out : out.subarray(0, offset);
|
||||
} finally { await handle.close(); }
|
||||
}
|
||||
}
|
||||
|
||||
const readers = new WeakMap<object, FileRangeReader>();
|
||||
|
||||
/** The ABIs an APK ships native libraries for (`lib/<abi>/`), without reading its manifest. */
|
||||
export async function apkAbis(path: string): Promise<string[]> {
|
||||
return [...new Set((await zipEntries(path)).map(entry => /^lib\/([^/]+)\//.exec(entry.filename)?.[1]).filter((abi): abi is string => !!abi))].sort();
|
||||
}
|
||||
|
||||
async function zipEntries(path: string): Promise<ZipEntry[]> {
|
||||
configure({ useWebWorkers: false });
|
||||
const reader = new FileRangeReader(path);
|
||||
const zip = new ZipReader(reader, { useWebWorkers: false });
|
||||
try {
|
||||
const entries = await zip.getEntries();
|
||||
if (entries.length > MAX_ARCHIVE_ENTRIES) throw archiveError("ARCHIVE_TOO_LARGE", "The archive has too many entries.", "Pass a smaller archive.");
|
||||
for (const entry of entries) readers.set(entry, reader);
|
||||
return entries;
|
||||
} catch (error) {
|
||||
if (error instanceof CiError) throw error;
|
||||
throw archiveError("ARCHIVE_INVALID", `${basename(path)} is not a readable ZIP archive: ${error instanceof Error ? error.message : String(error)}`, "Pass a valid .ipa or .apk.");
|
||||
}
|
||||
}
|
||||
|
||||
async function readZipEntry(entry: ZipEntry, maxBytes = MAX_METADATA_BYTES): Promise<Uint8Array> {
|
||||
if (entry.directory || !entry.getData) throw archiveError("ARCHIVE_INVALID", `${entry.filename} is not a file entry.`, "Pass a valid archive.");
|
||||
if (entry.uncompressedSize > maxBytes) throw archiveError("ARCHIVE_INVALID", `${entry.filename} is too large.`, "Pass a valid archive.");
|
||||
return entry.getData(new Uint8ArrayWriter(), { checkSignature: true, useWebWorkers: false });
|
||||
}
|
||||
|
||||
// ---- plist --------------------------------------------------------------------------------------
|
||||
|
||||
/** Property list reader for Info.plist: XML (Xcode default) or binary (`bplist00`). */
|
||||
|
||||
+24
-162
@@ -2,11 +2,12 @@ import { createHash, randomUUID } from "node:crypto";
|
||||
import { lstat, mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { createCiAgentSession, type CiAgentSession, type CiAgentSessionConfig } from "../capture/agent/session.js";
|
||||
import { createLocalCiAgentSession } from "../capture/agent/session-local.js";
|
||||
import type { CiAgentSession, CiAgentSessionConfig } from "../capture/agent/session.js";
|
||||
import type { CiAgentProviderName } from "../capture/agent/provider.js";
|
||||
import { assertReplayFits, createReplaySession, parseReplayPlan, replaySetup, type ReplayPlan } from "./replay.js";
|
||||
import { assertFlowsFit, createFlowSession, flowGoals, flowSetup, flowsNeedModel, parseFlowPlan, type FlowPlan } from "./flows.js";
|
||||
import type { ReplayPlan } from "./replay.js";
|
||||
import type { FlowPlan } from "./flows.js";
|
||||
import { assertAppUrl } from "./app-url.js";
|
||||
import { chooseStrategy, startStrategy, strategyInputs, turnBudget, type ExplorationStrategy, type ModelAuth } from "./strategy.js";
|
||||
import { CONTROLLER_SESSION, baseEnvironment, isRecord } from "../capture/common.js";
|
||||
import { controllerData, defaultRunner, openedSimulatorDevice } from "../capture/controller.js";
|
||||
import type { CommandRequest, CommandResult } from "../capture/run.js";
|
||||
@@ -22,7 +23,7 @@ import { screenSpace } from "../capture/route-capture.js";
|
||||
import { approveSessionUrlSchemes, simulatorControlActions, type EasCommand } from "../capture/simulator-control.js";
|
||||
import { DEFAULT_ANDROID_DEVICE } from "../config/explore-config.js";
|
||||
import { DUO_PANELS } from "../contracts/capture.js";
|
||||
import { createExplorationInputs } from "../config/exploration-inputs.js";
|
||||
import type { createExplorationInputs } from "../config/exploration-inputs.js";
|
||||
import { ExploreStore, getCoverage } from "../map/store.js";
|
||||
import { whoamiAccounts } from "../report/preview-deploy.js";
|
||||
import { CiError, serializeCiError, toCiError } from "../shared/ci-errors.js";
|
||||
@@ -34,7 +35,6 @@ import { runBlackBoxExploration, type BlackBoxExplorerResult, type BlackBoxFrame
|
||||
|
||||
export const DEFAULT_IOS_DEVICE = "iPhone 17 Pro";
|
||||
export const DEFAULT_BINARY_PROJECT = "expo-explore";
|
||||
export const DEFAULT_MODELS: Record<CiAgentProviderName, string> = { anthropic: "claude-sonnet-4-5", openai: "gpt-6.1-sol" };
|
||||
export const BINARY_LIMITS = { maxDurationMinutes: 15, maxScreens: 40, maxActions: 150 } as const;
|
||||
|
||||
export type BinaryExploreOptions = {
|
||||
@@ -76,7 +76,7 @@ export type BinaryExploreResult = {
|
||||
archive: Omit<InspectedArchive, "path"> & { path: string };
|
||||
session: { id: string; platform: Platform; device: string; owner: string; project: string };
|
||||
runtime: ActiveSimulatorRuntime;
|
||||
model: { provider: CiAgentProviderName | "replay" | "flows"; model: string; auth: "local-login" | "api-key" | "none" };
|
||||
model: Pick<ExplorationStrategy, "provider" | "model" | "auth">;
|
||||
exploration: BlackBoxExplorerResult;
|
||||
durationMs: number;
|
||||
};
|
||||
@@ -166,23 +166,6 @@ export function resolveSessionApp(runner: typeof defaultRunner, connection: Simu
|
||||
"Pass --bundle-id with the app's bundle identifier (iOS) or package name (Android).", { installed });
|
||||
}
|
||||
|
||||
/**
|
||||
* App-test inputs: each {{secret:NAME}} in the hints names an environment variable whose value Explore types for the
|
||||
* model. Read before any session starts, so a missing one costs nothing.
|
||||
*/
|
||||
export function hintInputs(hints: string | undefined, environment: NodeJS.ProcessEnv, replay?: ReplayPlan, flows?: FlowPlan, platform?: Platform): ReturnType<typeof createExplorationInputs> {
|
||||
const setup = replay ? replaySetup(replay) : flows ? flowSetup(flows, platform) : undefined;
|
||||
try { return createExplorationInputs({ instructions: [hints ?? "", ...(flows ? flowGoals(flows, platform) : [])].join("\n"), ...(setup ? { setup } : {}), environment }); }
|
||||
catch (error) {
|
||||
if (!(error instanceof CiError) || !error.code.startsWith("EXPLORATION_SECRET_")) throw error;
|
||||
const name = typeof error.details?.secretName === "string" ? error.details.secretName : "NAME";
|
||||
throw new CiError({ code: error.code, stage: "preflight", message: error.message, ...(error.details ? { details: error.details } : {}),
|
||||
nextAction: error.code === "EXPLORATION_SECRET_REFERENCE_INVALID"
|
||||
? "Write each input in --hints as {{secret:NAME}} with an uppercase environment variable name that is not a provider, Expo, or CI credential."
|
||||
: `Set ${name} in the environment that runs Explore (export ${name}=…) to the single-line test value, then rerun.` });
|
||||
}
|
||||
}
|
||||
|
||||
function validatedLimits(options: Pick<BinaryExploreOptions, "maxDurationMinutes" | "maxScreens" | "maxActions">) {
|
||||
const limits = { maxDurationMinutes: options.maxDurationMinutes ?? BINARY_LIMITS.maxDurationMinutes, maxScreens: options.maxScreens ?? BINARY_LIMITS.maxScreens, maxActions: options.maxActions ?? BINARY_LIMITS.maxActions };
|
||||
for (const [name, value] of Object.entries(limits)) if (!Number.isSafeInteger(value) || value < 1 || value > 1000) throw binaryError("EXPLORE_LIMIT_INVALID", `${name} must be an integer between 1 and 1000.`, "Correct the budget flag and rerun.");
|
||||
@@ -193,108 +176,6 @@ function binaryError(code: string, message: string, nextAction: string, details?
|
||||
return new CiError({ code, stage: "session", message, nextAction, ...(details ? { details } : {}) });
|
||||
}
|
||||
|
||||
type ModelAuth = "local-login" | "api-key";
|
||||
type ModelSessionOptions = Pick<BinaryExploreOptions, "provider" | "model" | "createSession">;
|
||||
|
||||
function modelSession(provider: CiAgentProviderName, auth: ModelAuth, options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
|
||||
const config = { provider, model: options.model ?? DEFAULT_MODELS[provider] };
|
||||
const session = options.createSession ? options.createSession(config, { env: environment, auth })
|
||||
: auth === "api-key" ? createCiAgentSession(config, { env: environment }) : createLocalCiAgentSession(config, { env: environment });
|
||||
return { session, provider, model: config.model, auth };
|
||||
}
|
||||
|
||||
/** The turns the model gets: three for each action and two for each screen, plus a few to look around and finish. */
|
||||
const turnBudget = (limits: { maxScreens: number; maxActions: number }): number => limits.maxActions * 3 + limits.maxScreens * 2 + 20;
|
||||
|
||||
/**
|
||||
* A replay plan is checked here as well as by the CLI, for callers that pass one directly. It takes the model's place, so
|
||||
* the options that only steer a model would do nothing; the plan's own fill and type steps name the secrets it types.
|
||||
*/
|
||||
function replayPlan(options: Pick<BinaryExploreOptions, "replay" | "provider" | "model" | "hints">): ReplayPlan | undefined {
|
||||
if (!options.replay) return undefined;
|
||||
const ignored = (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
|
||||
if (ignored.length) {
|
||||
throw binaryError("REPLAY_OPTIONS_IGNORED", `--replay takes the model's place, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
|
||||
`Drop ${ignored.map(name => `--${name}`).join(" and ")}. A {{secret:NAME}} in the plan's fill and type steps is read from the environment, as one in --hints is.`);
|
||||
}
|
||||
const plan = parseReplayPlan(options.replay);
|
||||
for (const [index, step] of plan.steps.entries()) {
|
||||
if (!("action" in step) || step.action !== "open") continue;
|
||||
try { assertAppUrl(step.url); }
|
||||
catch (error) { throw new CiError({ code: "REPLAY_PLAN_INVALID", stage: "exploration", message: `Step ${index + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the replay plan." }); }
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
|
||||
/** A replay plan takes the model's place: no model access is checked or used. */
|
||||
function replayModel(plan: ReplayPlan, platform: Platform, limits: { maxScreens: number; maxActions: number }) {
|
||||
assertReplayFits(plan, platform, { ...limits, maxTurns: turnBudget(limits) });
|
||||
const session = createReplaySession(plan, platform);
|
||||
return { session, budgetNote: session.budgetNote, provider: "replay" as const, model: "replay plan", auth: "none" as const };
|
||||
}
|
||||
|
||||
/**
|
||||
* A flows file is checked here as well as by the CLI. It decides every step, as a replay plan does, but a step it
|
||||
* describes in words is carried out by a model, so --provider, --model and --hints apply only when it has one.
|
||||
*/
|
||||
function flowsPlan(options: Pick<BinaryExploreOptions, "flows" | "replay" | "provider" | "model" | "hints">): FlowPlan | undefined {
|
||||
if (!options.flows) return undefined;
|
||||
if (options.replay) throw binaryError("FLOWS_WITH_REPLAY", "--flows and --replay each decide every step, so only one of them can run.", "Drop --replay, or move its steps into a flow.");
|
||||
const plan = parseFlowPlan(options.flows);
|
||||
const ignored = flowsNeedModel(plan) ? [] : (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
|
||||
if (ignored.length) {
|
||||
throw binaryError("FLOWS_OPTIONS_IGNORED", `Every step of the flows file is exact, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
|
||||
`Drop ${ignored.map(name => `--${name}`).join(" and ")}, or describe a step in words ({"do": "…"}) for a model to carry out.`);
|
||||
}
|
||||
for (const [index, flow] of plan.flows.entries()) for (const [stepIndex, step] of flow.steps.entries()) {
|
||||
if (!("action" in step) || step.action !== "open") continue;
|
||||
try { assertAppUrl(step.url); }
|
||||
catch (error) { throw new CiError({ code: "FLOWS_PLAN_INVALID", stage: "exploration", message: `Flow ${index + 1} (${flow.name}), step ${stepIndex + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the flows file." }); }
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
|
||||
/** The flows run in place of a model's exploration; a model is checked, and used, only when a step is described in words. */
|
||||
async function flowsModel(plan: FlowPlan, platform: Platform, limits: { maxScreens: number; maxActions: number }, options: ModelSessionOptions & Pick<BinaryExploreOptions, "hints">,
|
||||
environment: NodeJS.ProcessEnv, say: (message: string) => void, app: { name: string }) {
|
||||
assertFlowsFit(plan, platform, { ...limits, maxTurns: turnBudget(limits) });
|
||||
const goals = flowsNeedModel(plan, platform) ? await preflightModelSession(options, environment, say) : undefined;
|
||||
const session = createFlowSession(plan, platform, { app, ...(goals ? { model: goals.session } : {}), ...(options.hints ? { hints: options.hints } : {}) });
|
||||
return { session, budgetNote: session.budgetNote, flows: session.notReached, provider: "flows" as const, model: goals ? `${goals.provider} ${goals.model}` : session.model, auth: goals?.auth ?? "none" as const };
|
||||
}
|
||||
|
||||
/** Local `claude`/`codex` login unless an API key is present; preflight proves access before a simulator is paid for. */
|
||||
export function selectModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
|
||||
const provider: CiAgentProviderName = options.provider ?? (environment.OPENAI_API_KEY?.trim() && !environment.ANTHROPIC_API_KEY?.trim() ? "openai" : "anthropic");
|
||||
const keyName = provider === "anthropic" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY";
|
||||
return modelSession(provider, environment[keyName]?.trim() ? "api-key" : "local-login", options, environment);
|
||||
}
|
||||
|
||||
/**
|
||||
* With --provider or --model, only that choice is tried. Otherwise every model access this shell has is tried
|
||||
* in turn (API keys, then the local claude and codex logins) and the first that passes preflight runs.
|
||||
*/
|
||||
export async function preflightModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv, say: (message: string) => void) {
|
||||
const candidates = options.provider || options.model ? [selectModelSession(options, environment)] : [
|
||||
...(["anthropic", "openai"] as const).filter(provider => environment[provider === "anthropic" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY"]?.trim())
|
||||
.map(provider => modelSession(provider, "api-key", options, environment)),
|
||||
modelSession("anthropic", "local-login", options, environment),
|
||||
modelSession("openai", "local-login", options, environment),
|
||||
];
|
||||
let firstError: unknown;
|
||||
for (const [index, candidate] of candidates.entries()) {
|
||||
say(`checking ${candidate.provider} ${candidate.model} (${candidate.auth === "api-key" ? "API key" : "local login"})`);
|
||||
try {
|
||||
await candidate.session.preflight();
|
||||
return candidate;
|
||||
} catch (error) {
|
||||
firstError ??= error;
|
||||
if (index < candidates.length - 1) say(`${candidate.provider} ${candidate.auth === "api-key" ? "API key" : "local login"} unavailable (${error instanceof Error ? error.message : String(error)}); trying the next`);
|
||||
}
|
||||
}
|
||||
throw firstError;
|
||||
}
|
||||
|
||||
export type SessionExploreOptions = Omit<BinaryExploreOptions, "archive" | "bundleId" | "owner" | "project"> & {
|
||||
/** App to explore; defaults to the app in the foreground, else the only user-installed app. */
|
||||
bundleId?: string;
|
||||
@@ -329,8 +210,7 @@ export type ConnectedSessionExploration = {
|
||||
limits: { maxDurationMinutes: number; maxScreens: number; maxActions: number };
|
||||
store: ExploreStore;
|
||||
map: ExploreMap;
|
||||
/** `flows`: the session takes the explorer's flow tools, and names the checkpoints it never attempted. */
|
||||
model: { session: Pick<CiAgentSession, "run">; budgetNote?: () => string | undefined; flows?: () => Array<{ flow: string; checkpoints: string[] }> };
|
||||
strategy: ExplorationStrategy;
|
||||
/** `eas simulator:get --json` for the attached session; descriptive (OS version) only. */
|
||||
sessionJson: () => string;
|
||||
/** agent-device session name on the daemon; explore:session reuses the caller's session so the device is not "in use" by another. */
|
||||
@@ -353,7 +233,7 @@ export type ConnectedSessionExploration = {
|
||||
|
||||
/** Drives an agent-device daemon we already hold a connection to: optional install, open, then the model-led exploration. */
|
||||
export async function exploreConnectedSession(ctx: ConnectedSessionExploration): Promise<{ runtime: ActiveSimulatorRuntime; exploration: BlackBoxExplorerResult }> {
|
||||
const { connection, platform, device, bundleId, app, install, work, runner, environment, secrets, inputs, say, now, limits, store, map, model, sessionJson, launchUrl, hints, allowDestructive, local } = ctx;
|
||||
const { connection, platform, device, bundleId, app, install, work, runner, environment, secrets, inputs, say, now, limits, store, map, strategy, sessionJson, launchUrl, hints, allowDestructive, local } = ctx;
|
||||
const controllerSession = ctx.controllerSession ?? CONTROLLER_SESSION;
|
||||
const controllerKind = connectionController(connection);
|
||||
const controllerName = controllerKind === "argent" ? "Argent" : "agent-device";
|
||||
@@ -392,7 +272,7 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
|
||||
ctx.afterInstall?.();
|
||||
// On a local device, the first open also builds agent-device's iOS runner.
|
||||
// Flows start from the app's first screen: an app already running on an attached session is relaunched, not brought forward.
|
||||
const opened = controller(["open", bundleId, ...(model.flows ? ["--relaunch"] : []), "--level", "full", "--json"], local ? 15 * 60_000 : 2 * 60_000);
|
||||
const opened = controller(["open", bundleId, ...(strategy.flows ? ["--relaunch"] : []), "--level", "full", "--json"], local ? 15 * 60_000 : 2 * 60_000);
|
||||
const openedDevice = openedSimulatorDevice(opened.stdout, bundleId, "initial app open", platform, controllerSession);
|
||||
const data = controllerData(opened.stdout, "initial app open");
|
||||
const deviceData = data.device && typeof data.device === "object" ? data.device as Record<string, unknown> : {};
|
||||
@@ -539,12 +419,12 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
|
||||
map.gaps.push({ id: `gap_${randomUUID().slice(0, 8)}`, kind: "coverage", status: "open", message: reason, createdAt: new Date(now()).toISOString() });
|
||||
say("exploration blocked: the app does not stay running");
|
||||
blockedAtLaunch = { stopReason: "blocked", summary: reason, screens: 0, observations: 0, actions: 0, turns: 0,
|
||||
...(model.flows ? { flows: model.flows().map(({ flow, checkpoints }) => ({ name: flow, status: "failed" as const, reason, checkpoints: checkpoints.map(name => ({ name, missing: "The app did not stay open." })) })) } : {}) };
|
||||
...(strategy.flows ? { flows: strategy.flows().map(({ flow, checkpoints }) => ({ name: flow, status: "failed" as const, reason, checkpoints: checkpoints.map(name => ({ name, missing: "The app did not stay open." })) })) } : {}) };
|
||||
}
|
||||
}
|
||||
const exploration = blockedAtLaunch ?? await runBlackBoxExploration({ session: model.session, store, map, app: { name: app.name, bundleId, platform, ...(app.version ? { version: app.version } : {}) },
|
||||
const exploration = blockedAtLaunch ?? await runBlackBoxExploration({ session: strategy.session, store, map, app: { name: app.name, bundleId, platform, ...(app.version ? { version: app.version } : {}) },
|
||||
inputs, ...(hints ? { hints: hints } : {}), ...(launchUrl ? { launchUrl: launchUrl } : {}), ...(allowDestructive ? { allowDestructive: allowDestructive } : {}), ...(ctx.appIcon ? { appIcon: ctx.appIcon } : {}),
|
||||
...(model.budgetNote ? { budgetNote: model.budgetNote } : {}), ...(model.flows ? { flows: { restart, notReached: model.flows } } : {}),
|
||||
...(strategy.budgetNote ? { budgetNote: strategy.budgetNote } : {}), ...(strategy.flows ? { flows: { restart, notReached: strategy.flows } } : {}),
|
||||
deadlineMs, maxScreens: limits.maxScreens, maxActions: limits.maxActions, maxTurns: turnBudget(limits), now, observe, progress: say,
|
||||
act: async action => {
|
||||
const perform = async () => {
|
||||
@@ -609,11 +489,10 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
|
||||
const environment = options.environment ?? process.env;
|
||||
const runner = options.runner ?? defaultRunner;
|
||||
const limits = validatedLimits(options);
|
||||
const replay = replayPlan(options);
|
||||
const flows = flowsPlan(options);
|
||||
const choice = chooseStrategy(options);
|
||||
// A flows run without --platform checks its secrets once the session reports its platform, before any device work, so a
|
||||
// secret that only another platform's steps type is not required.
|
||||
let inputs = flows && !options.platform ? undefined : hintInputs(options.hints, environment, replay, flows, options.platform);
|
||||
let inputs = choice.kind === "flows" && !options.platform ? undefined : strategyInputs(choice, options.hints, environment, options.platform);
|
||||
if (options.bundleId !== undefined && !appIdPattern.test(options.bundleId)) throw binaryError("BUNDLE_ID_INVALID", `${options.bundleId} is not a bundle or package id.`, "Pass --bundle-id with the app's bundle identifier (iOS) or package name (Android).");
|
||||
let connection: SimulatorConnection;
|
||||
try { connection = await resolveAttachedSimulatorSession(environment, options.envFile, options.session); }
|
||||
@@ -659,7 +538,7 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
|
||||
}
|
||||
platform = info.platform;
|
||||
if (!inputs) {
|
||||
inputs = hintInputs(options.hints, environment, replay, flows, platform);
|
||||
inputs = strategyInputs(choice, options.hints, environment, platform);
|
||||
secrets.push(...inputs.values);
|
||||
}
|
||||
const candidates = daemonSessions(runner, connection, work, platform, environment).filter(item => !device || item.device === device);
|
||||
@@ -681,16 +560,15 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
|
||||
map.run.environment = { deviceModel: device };
|
||||
session = { id: connection.sessionId, platform, device, name: info.name, ...(info.deviceRunSessionUrl ? { url: info.deviceRunSessionUrl } : {}) };
|
||||
|
||||
const model = replay ? replayModel(replay, platform, limits) : flows ? await flowsModel(flows, platform, limits, options, environment, say, app)
|
||||
: await preflightModelSession(options, environment, say);
|
||||
const strategy = await startStrategy(choice, { platform, limits, options, environment, say, app });
|
||||
const outputDirectory = await realpath(await mkdir(options.output, { recursive: true }).then(() => options.output));
|
||||
const store = new ExploreStore(outputDirectory);
|
||||
if (await store.exists()) throw binaryError("OUTPUT_NOT_EMPTY", `${outputDirectory} already holds an Explore map.`, "Pass an empty --output directory.");
|
||||
|
||||
({ runtime, exploration } = await exploreConnectedSession({ connection, platform, device, bundleId: app.bundleId, app, work, runner, environment, secrets, inputs, say, now, limits, store, map, model, sessionJson,
|
||||
({ runtime, exploration } = await exploreConnectedSession({ connection, platform, device, bundleId: app.bundleId, app, work, runner, environment, secrets, inputs, say, now, limits, store, map, strategy, sessionJson,
|
||||
...(existing ? { controllerSession: existing.name } : {}),
|
||||
...(options.launchUrl ? { launchUrl: options.launchUrl } : {}), ...(options.hints ? { hints: options.hints } : {}), ...(options.allowDestructive ? { allowDestructive: options.allowDestructive } : {}) }));
|
||||
return { outputDirectory, mapPath: store.mapPath, app: { ...app, platform }, session, runtime, model: { provider: model.provider, model: model.model, auth: model.auth }, exploration, durationMs: Date.now() - startedAt };
|
||||
return { outputDirectory, mapPath: store.mapPath, app: { ...app, platform }, session, runtime, model: { provider: strategy.provider, model: strategy.model, auth: strategy.auth }, exploration, durationMs: Date.now() - startedAt };
|
||||
} catch (error) {
|
||||
primaryError = toCiError(error, { code: "EXPLORE_SESSION_FAILED", stage: "exploration", nextAction: "Inspect the diagnostic, correct the failing prerequisite, and rerun explore:session. The session was left running." });
|
||||
throw primaryError;
|
||||
@@ -775,11 +653,10 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
|
||||
const environment = options.environment ?? process.env;
|
||||
const runner = options.runner ?? defaultRunner;
|
||||
const limits = validatedLimits(options);
|
||||
const replay = replayPlan(options);
|
||||
const flows = flowsPlan(options);
|
||||
const choice = chooseStrategy(options);
|
||||
// The archive's name gives its platform before it is read (inspectArchive confirms it), so a secret only another platform's steps type is not required.
|
||||
const namedPlatform = options.platform ?? (() => { try { return archiveKind(options.archive) === "android-apk" ? "android" as const : "ios" as const; } catch { return undefined; } })();
|
||||
const inputs = hintInputs(options.hints, environment, replay, flows, namedPlatform);
|
||||
const inputs = strategyInputs(choice, options.hints, environment, namedPlatform);
|
||||
const local = options.simulator === "local";
|
||||
// A local emulator runs this Mac's ABI (arm64-v8a on Apple silicon); EAS's runs x86_64.
|
||||
const emulatorAbi = local ? hostEmulatorAbi() : "x86_64";
|
||||
@@ -796,8 +673,7 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
|
||||
say(`${archive.appName} ${archive.version ?? ""} (${bundleId}, ${platform}, ${(archive.bytes / 1048576).toFixed(0)} MB, sha256 ${archive.sha256.slice(0, 12)})`);
|
||||
if (archive.missingLibs?.length && emulatorAbi === "x86_64") say(`warning: lib/x86_64 lacks ${archive.missingLibs.slice(0, 4).join(", ")}${archive.missingLibs.length > 4 ? ` and ${archive.missingLibs.length - 4} more` : ""} that another ABI ships; the app may stop with UnsatisfiedLinkError on the emulator`);
|
||||
|
||||
const model = replay ? replayModel(replay, platform, limits) : flows ? await flowsModel(flows, platform, limits, options, environment, say, { name: archive.appName })
|
||||
: await preflightModelSession(options, environment, say);
|
||||
const strategy = await startStrategy(choice, { platform, limits, options, environment, say, app: { name: archive.appName } });
|
||||
|
||||
const outputDirectory = await realpath(await mkdir(options.output, { recursive: true }).then(() => options.output));
|
||||
const store = new ExploreStore(outputDirectory);
|
||||
@@ -837,7 +713,7 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
|
||||
...(appIcon ? { appIcon } : {}), app: { name: archive.appName, ...(archive.version ? { version: archive.version } : {}) },
|
||||
// A local device has the app installed by simctl or adb before the controller starts.
|
||||
...(localDevice ? { local: localDevice } : { install: { path: installPath, kind: archive.kind } }),
|
||||
work, runner, environment, secrets: [environment.EXPO_TOKEN ?? "", connection.daemonAuthToken, ...inputs.values].filter(Boolean), inputs, say, now, limits, store, map, model,
|
||||
work, runner, environment, secrets: [environment.EXPO_TOKEN ?? "", connection.daemonAuthToken, ...inputs.values].filter(Boolean), inputs, say, now, limits, store, map, strategy,
|
||||
urlSchemes: archive.urlSchemes, sessionJson,
|
||||
...(options.launchUrl ? { launchUrl: options.launchUrl } : {}), ...(options.hints ? { hints: options.hints } : {}), ...(options.allowDestructive ? { allowDestructive: options.allowDestructive } : {}) });
|
||||
if (local) {
|
||||
@@ -920,7 +796,7 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
|
||||
}
|
||||
if (primaryError) throw primaryError;
|
||||
return { outputDirectory, mapPath: store.mapPath, archive, session: { id: sessionId!, platform, device, owner, project }, runtime: runtime!,
|
||||
model: { provider: model.provider, model: model.model, auth: model.auth }, exploration: exploration!, durationMs: Date.now() - startedAt };
|
||||
model: { provider: strategy.provider, model: strategy.model, auth: strategy.auth }, exploration: exploration!, durationMs: Date.now() - startedAt };
|
||||
}
|
||||
|
||||
function lastLine(text: string): string { return text.trim().split(/\r?\n/).filter(Boolean).at(-1) ?? ""; }
|
||||
@@ -935,10 +811,6 @@ async function easLoggedIn(runner: typeof defaultRunner, environment: NodeJS.Pro
|
||||
} catch { return false; }
|
||||
}
|
||||
|
||||
/** Schemes the OS routes to a browser, the store, Settings, or another app: opening one hands the session away from the app under test. */
|
||||
const SYSTEM_URL_SCHEMES = /^(https?|ftp|file|javascript|data|blob|mailto|tel|telprompt|facetime(-audio)?|sms|mms|imessage|itms(-apps|-services|-beta)?|macappstore|app-settings|prefs|shortcuts|maps|music|videos|photos-redirect|x-apple-[a-z-]+|market|intent|content|android-app|geo|package):$/i;
|
||||
|
||||
/** The model may only open app-owned URLs; an https page or a store link would hand the session to another app. */
|
||||
const RUNNER_BUSY = /still finishing a previous command/i;
|
||||
const sleepSync = (ms: number): void => { Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); };
|
||||
|
||||
@@ -957,16 +829,6 @@ export function retryWhileBusy<T>(run: () => T, pause: (ms: number) => void, bud
|
||||
}
|
||||
}
|
||||
|
||||
export function assertAppUrl(url: string, appSchemes: readonly string[] = []): void {
|
||||
let parsed: URL;
|
||||
try { parsed = new URL(url); } catch { throw binaryError("AGENT_URL_INVALID", `Not a URL: ${url}`, "Pass a custom-scheme app URL."); }
|
||||
if (SYSTEM_URL_SCHEMES.test(parsed.protocol)) throw binaryError("AGENT_URL_NOT_ALLOWED", `${parsed.protocol} URLs leave the app; only the app's own URL schemes are allowed.`, "Use the app's own URL scheme.");
|
||||
const scheme = parsed.protocol.slice(0, -1).toLowerCase();
|
||||
if (appSchemes.length && !appSchemes.includes(scheme)) {
|
||||
throw binaryError("AGENT_URL_NOT_ALLOWED", `${parsed.protocol} is not a URL scheme this app registers (${appSchemes.join(", ")}).`, "Use one of the app's own URL schemes.");
|
||||
}
|
||||
}
|
||||
|
||||
export const BLANK_SCREENSHOT_MESSAGE = "The app blocks screen capture on this screen (Android FLAG_SECURE), so its screenshot came back blank.";
|
||||
|
||||
export { solidColour };
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
import { createCiAgentSession, type CiAgentSession } from "../capture/agent/session.js";
|
||||
import { createLocalCiAgentSession } from "../capture/agent/session-local.js";
|
||||
import type { CiAgentProviderName } from "../capture/agent/provider.js";
|
||||
import { createExplorationInputs } from "../config/exploration-inputs.js";
|
||||
import { CiError } from "../shared/ci-errors.js";
|
||||
import type { Platform } from "../types.js";
|
||||
import { assertAppUrl } from "./app-url.js";
|
||||
import { assertFlowsFit, createFlowSession, flowGoals, flowSetup, flowsNeedModel, parseFlowPlan, type FlowPlan } from "./flows.js";
|
||||
import { assertReplayFits, createReplaySession, parseReplayPlan, replaySetup, type ReplayPlan } from "./replay.js";
|
||||
import type { BinaryExploreOptions } from "./run.js";
|
||||
|
||||
/**
|
||||
* How an exploration decides what to do on the device. The explorer (`runBlackBoxExploration`) owns the device and the
|
||||
* tools that drive it, the budgets, what is refused, and what is recorded; a strategy only calls those tools, through
|
||||
* `session.run`. A model explores on its own, a replay plan takes fixed steps, and a flows file takes named flows.
|
||||
* A new strategy is one more case here: what `chooseStrategy` checks before any device starts, and the session
|
||||
* `startStrategy` runs.
|
||||
*/
|
||||
export type StrategyChoice = { kind: "model" } | { kind: "replay"; plan: ReplayPlan } | { kind: "flows"; plan: FlowPlan };
|
||||
|
||||
export type ModelAuth = "local-login" | "api-key";
|
||||
|
||||
/** A strategy ready to run, and how the report names it. */
|
||||
export type ExplorationStrategy = {
|
||||
session: Pick<CiAgentSession, "run">;
|
||||
provider: CiAgentProviderName | "replay" | "flows";
|
||||
model: string;
|
||||
auth: ModelAuth | "none";
|
||||
/** A sentence a budget stop's gap adds to its own: the planned screens or checkpoints the run did not reach. */
|
||||
budgetNote?: () => string | undefined;
|
||||
/**
|
||||
* Takes the explorer's flow tools (start_flow, checkpoint, end_flow), with a relaunch before each flow after the first,
|
||||
* and names each flow's checkpoints it never attempted, for the explorer to record as missing.
|
||||
*/
|
||||
flows?: () => Array<{ flow: string; checkpoints: string[] }>;
|
||||
};
|
||||
|
||||
export const DEFAULT_MODELS: Record<CiAgentProviderName, string> = { anthropic: "claude-sonnet-4-5", openai: "gpt-6.1-sol" };
|
||||
|
||||
type StrategyOptions = Pick<BinaryExploreOptions, "replay" | "flows" | "provider" | "model" | "hints" | "createSession">;
|
||||
type ModelSessionOptions = Pick<BinaryExploreOptions, "provider" | "model" | "createSession">;
|
||||
|
||||
function strategyError(code: string, message: string, nextAction: string): CiError {
|
||||
return new CiError({ code, stage: "session", message, nextAction });
|
||||
}
|
||||
|
||||
/** The turns the model gets: three for each action and two for each screen, plus a few to look around and finish. */
|
||||
export const turnBudget = (limits: { maxScreens: number; maxActions: number }): number => limits.maxActions * 3 + limits.maxScreens * 2 + 20;
|
||||
|
||||
/**
|
||||
* The strategy the options ask for, checked before any device starts. A plan file is checked here as well as by the
|
||||
* CLI, for callers that pass one directly.
|
||||
*/
|
||||
export function chooseStrategy(options: StrategyOptions): StrategyChoice {
|
||||
if (options.replay && options.flows) throw strategyError("FLOWS_WITH_REPLAY", "--flows and --replay each decide every step, so only one of them can run.", "Drop --replay, or move its steps into a flow.");
|
||||
if (options.replay) return { kind: "replay", plan: replayPlan(options.replay, options) };
|
||||
if (options.flows) return { kind: "flows", plan: flowsPlan(options.flows, options) };
|
||||
return { kind: "model" };
|
||||
}
|
||||
|
||||
/**
|
||||
* A replay plan takes the model's place, so the options that only steer a model would do nothing; the plan's own fill
|
||||
* and type steps name the secrets it types.
|
||||
*/
|
||||
function replayPlan(value: ReplayPlan, options: StrategyOptions): ReplayPlan {
|
||||
const ignored = (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
|
||||
if (ignored.length) {
|
||||
throw strategyError("REPLAY_OPTIONS_IGNORED", `--replay takes the model's place, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
|
||||
`Drop ${ignored.map(name => `--${name}`).join(" and ")}. A {{secret:NAME}} in the plan's fill and type steps is read from the environment, as one in --hints is.`);
|
||||
}
|
||||
const plan = parseReplayPlan(value);
|
||||
for (const [index, step] of plan.steps.entries()) {
|
||||
if (!("action" in step) || step.action !== "open") continue;
|
||||
try { assertAppUrl(step.url); }
|
||||
catch (error) { throw new CiError({ code: "REPLAY_PLAN_INVALID", stage: "exploration", message: `Step ${index + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the replay plan." }); }
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
|
||||
/**
|
||||
* A flows file decides every step, as a replay plan does, but a step it describes in words is carried out by a model,
|
||||
* so --provider, --model and --hints apply only when it has one.
|
||||
*/
|
||||
function flowsPlan(value: FlowPlan, options: StrategyOptions): FlowPlan {
|
||||
const plan = parseFlowPlan(value);
|
||||
const ignored = flowsNeedModel(plan) ? [] : (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
|
||||
if (ignored.length) {
|
||||
throw strategyError("FLOWS_OPTIONS_IGNORED", `Every step of the flows file is exact, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
|
||||
`Drop ${ignored.map(name => `--${name}`).join(" and ")}, or describe a step in words ({"do": "…"}) for a model to carry out.`);
|
||||
}
|
||||
for (const [index, flow] of plan.flows.entries()) for (const [stepIndex, step] of flow.steps.entries()) {
|
||||
if (!("action" in step) || step.action !== "open") continue;
|
||||
try { assertAppUrl(step.url); }
|
||||
catch (error) { throw new CiError({ code: "FLOWS_PLAN_INVALID", stage: "exploration", message: `Flow ${index + 1} (${flow.name}), step ${stepIndex + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the flows file." }); }
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
|
||||
/**
|
||||
* App-test inputs: each {{secret:NAME}} in the hints, and in the strategy's own fill and type steps and described steps,
|
||||
* names an environment variable whose value Explore types. Read before any session starts, so a missing one costs
|
||||
* nothing; a step limited to another platform than `platform`, when it is known, needs none.
|
||||
*/
|
||||
export function strategyInputs(choice: StrategyChoice, hints: string | undefined, environment: NodeJS.ProcessEnv, platform?: Platform): ReturnType<typeof createExplorationInputs> {
|
||||
const setup = choice.kind === "replay" ? replaySetup(choice.plan) : choice.kind === "flows" ? flowSetup(choice.plan, platform) : undefined;
|
||||
try { return createExplorationInputs({ instructions: [hints ?? "", ...(choice.kind === "flows" ? flowGoals(choice.plan, platform) : [])].join("\n"), ...(setup ? { setup } : {}), environment }); }
|
||||
catch (error) {
|
||||
if (!(error instanceof CiError) || !error.code.startsWith("EXPLORATION_SECRET_")) throw error;
|
||||
const name = typeof error.details?.secretName === "string" ? error.details.secretName : "NAME";
|
||||
throw new CiError({ code: error.code, stage: "preflight", message: error.message, ...(error.details ? { details: error.details } : {}),
|
||||
nextAction: error.code === "EXPLORATION_SECRET_REFERENCE_INVALID"
|
||||
? "Write each input in --hints as {{secret:NAME}} with an uppercase environment variable name that is not a provider, Expo, or CI credential."
|
||||
: `Set ${name} in the environment that runs Explore (export ${name}=…) to the single-line test value, then rerun.` });
|
||||
}
|
||||
}
|
||||
|
||||
/** The session the chosen strategy runs. Model access is checked here, before a simulator is paid for, and only when the strategy uses a model. */
|
||||
export async function startStrategy(choice: StrategyChoice, context: { platform: Platform; limits: { maxScreens: number; maxActions: number }; options: StrategyOptions;
|
||||
environment: NodeJS.ProcessEnv; say: (message: string) => void; app: { name: string } }): Promise<ExplorationStrategy> {
|
||||
const { platform, limits, options, environment, say, app } = context;
|
||||
if (choice.kind === "replay") {
|
||||
assertReplayFits(choice.plan, platform, { ...limits, maxTurns: turnBudget(limits) });
|
||||
const session = createReplaySession(choice.plan, platform);
|
||||
return { session, budgetNote: session.budgetNote, provider: "replay", model: "replay plan", auth: "none" };
|
||||
}
|
||||
if (choice.kind === "flows") {
|
||||
assertFlowsFit(choice.plan, platform, { ...limits, maxTurns: turnBudget(limits) });
|
||||
const goals = flowsNeedModel(choice.plan, platform) ? await preflightModelSession(options, environment, say) : undefined;
|
||||
const session = createFlowSession(choice.plan, platform, { app, ...(goals ? { model: goals.session } : {}), ...(options.hints ? { hints: options.hints } : {}) });
|
||||
return { session, budgetNote: session.budgetNote, flows: session.notReached, provider: "flows", model: goals ? `${goals.provider} ${goals.model}` : session.model, auth: goals?.auth ?? "none" };
|
||||
}
|
||||
return preflightModelSession(options, environment, say);
|
||||
}
|
||||
|
||||
function modelSession(provider: CiAgentProviderName, auth: ModelAuth, options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
|
||||
const config = { provider, model: options.model ?? DEFAULT_MODELS[provider] };
|
||||
const session = options.createSession ? options.createSession(config, { env: environment, auth })
|
||||
: auth === "api-key" ? createCiAgentSession(config, { env: environment }) : createLocalCiAgentSession(config, { env: environment });
|
||||
return { session, provider, model: config.model, auth };
|
||||
}
|
||||
|
||||
/** Local `claude`/`codex` login unless an API key is present; preflight proves access before a simulator is paid for. */
|
||||
export function selectModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
|
||||
const provider: CiAgentProviderName = options.provider ?? (environment.OPENAI_API_KEY?.trim() && !environment.ANTHROPIC_API_KEY?.trim() ? "openai" : "anthropic");
|
||||
const keyName = provider === "anthropic" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY";
|
||||
return modelSession(provider, environment[keyName]?.trim() ? "api-key" : "local-login", options, environment);
|
||||
}
|
||||
|
||||
/**
|
||||
* With --provider or --model, only that choice is tried. Otherwise every model access this shell has is tried
|
||||
* in turn (API keys, then the local claude and codex logins) and the first that passes preflight runs.
|
||||
*/
|
||||
export async function preflightModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv, say: (message: string) => void) {
|
||||
const candidates = options.provider || options.model ? [selectModelSession(options, environment)] : [
|
||||
...(["anthropic", "openai"] as const).filter(provider => environment[provider === "anthropic" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY"]?.trim())
|
||||
.map(provider => modelSession(provider, "api-key", options, environment)),
|
||||
modelSession("anthropic", "local-login", options, environment),
|
||||
modelSession("openai", "local-login", options, environment),
|
||||
];
|
||||
let firstError: unknown;
|
||||
for (const [index, candidate] of candidates.entries()) {
|
||||
say(`checking ${candidate.provider} ${candidate.model} (${candidate.auth === "api-key" ? "API key" : "local login"})`);
|
||||
try {
|
||||
await candidate.session.preflight();
|
||||
return candidate;
|
||||
} catch (error) {
|
||||
firstError ??= error;
|
||||
if (index < candidates.length - 1) say(`${candidate.provider} ${candidate.auth === "api-key" ? "API key" : "local login"} unavailable (${error instanceof Error ? error.message : String(error)}); trying the next`);
|
||||
}
|
||||
}
|
||||
throw firstError;
|
||||
}
|
||||
@@ -3,7 +3,7 @@ import { existsSync } from "node:fs";
|
||||
import { mkdir, readdir, readFile, rm, stat, symlink, writeFile } from "node:fs/promises";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { apkAbis } from "../binary/archive.js";
|
||||
import { apkAbis } from "../shared/archive-entries.js";
|
||||
import { exploreCacheRoot } from "../shared/cache-root.js";
|
||||
import { CiError } from "../shared/ci-errors.js";
|
||||
import { isRecord, parseJson } from "./common.js";
|
||||
|
||||
@@ -7,7 +7,7 @@ import { CiError, environmentSecrets } from "../shared/ci-errors.js";
|
||||
import { silentProgress, type Progress } from "../shared/progress.js";
|
||||
import { githubRunLink, safeText, silentTimings, type Timings } from "../shared/timings.js";
|
||||
import { retryGitHubLookup } from "../report/github-lookup.js";
|
||||
import { safeRelativePath } from "../binary/archive.js";
|
||||
import { safeRelativePath } from "../shared/archive-entries.js";
|
||||
import { CAPTURE_CACHE_MAX_BYTES, CAPTURE_CACHE_MAX_FILES } from "./artifact-cache.js";
|
||||
import { baselineLookup, githubContext, trustedBaselineRun, type GithubArtifact, type GithubRun } from "./workflow-steps.js";
|
||||
|
||||
|
||||
+2
-1
@@ -11,7 +11,7 @@ import { help } from "./cli/help.js";
|
||||
import { listScreens, getScreen, listElements, listObservations, getObservation, nextFrontier, listAttempts, getAttempt, listTransitions, getTransition, neighbors, pathQuery, listJourneys, getJourney, listFrontier, listGaps, summary, search, status, validate, exportArtifacts } from "./cli/map-commands.js";
|
||||
import { compareCommand, exportWeb, view } from "./cli/catalog-commands.js";
|
||||
import { ciReport, ciPublish, ciFailure, ciDeploy, ciPullRequest, ciFindBaseline, ciCompareRefs, ciAfterCapture, setup, doctor, sourceVerify, captureReview, captureBaseline, configPlatforms, configTargetList } from "./cli/ci-commands.js";
|
||||
import { exploreBinary, exploreSession } from "./cli/binary-commands.js";
|
||||
import { compareFlows, exploreBinary, exploreSession } from "./cli/binary-commands.js";
|
||||
import { reports, reportUpload } from "./cli/report-commands.js";
|
||||
|
||||
const handlers: Record<string, () => Promise<void> | void> = {
|
||||
@@ -57,6 +57,7 @@ const handlers: Record<string, () => Promise<void> | void> = {
|
||||
"config:targets": configTargetList,
|
||||
doctor,
|
||||
compare: compareCommand,
|
||||
"compare:flows": compareFlows,
|
||||
view,
|
||||
help,
|
||||
"--help": help,
|
||||
|
||||
@@ -19,6 +19,7 @@ const aliases: Record<string, string> = {
|
||||
"frontier list": "frontier:list",
|
||||
"gap list": "gap:list",
|
||||
"export web": "export:web",
|
||||
"compare flows": "compare:flows",
|
||||
"ci report": "ci:report",
|
||||
"ci publish": "ci:publish",
|
||||
"ci failure": "ci:failure",
|
||||
@@ -104,6 +105,7 @@ const flagKinds: Record<string, Record<string, "value" | "boolean">> = {
|
||||
"ci:find-baseline": { sha:"value", target:"value", platform:"value" },
|
||||
"ci:compare-refs": { project:"value", base:"value", head:"value", title:"value", output:"value" },
|
||||
"ci:after-capture": { output:"value", checkout:"value" },
|
||||
"compare:flows": { before: "value", after: "value", output: "value" },
|
||||
compare: {
|
||||
baseline: "value",
|
||||
current: "value",
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { stat } from "node:fs/promises";
|
||||
import { readFile, stat, writeFile } from "node:fs/promises";
|
||||
import { basename, join, resolve } from "node:path";
|
||||
import { DEFAULT_BINARY_PROJECT, runBinaryExploration, runSessionExploration } from "../binary/run.js";
|
||||
import { openViewerInBrowser, startViewerServer, waitForViewerShutdown } from "../map/viewer-server.js";
|
||||
@@ -6,10 +6,14 @@ import { writeExplorationReport } from "../binary/report.js";
|
||||
import { assertHostingExport, deployExplorationReport, zipReportDirectory } from "../report/preview-deploy.js";
|
||||
import { keepReports } from "./report-commands.js";
|
||||
import { warnIgnoredExpiry } from "../config/explore-config.js";
|
||||
import { argumentError, flag, flags, has, nonNegativeInteger, oneOf, output, positional, positiveInteger } from "./args.js";
|
||||
import { argumentError, flag, flags, has, nonNegativeInteger, oneOf, output, positional, positiveInteger, required } from "./args.js";
|
||||
import { readReplayPlan } from "../binary/replay.js";
|
||||
import { readFlowPlan } from "../binary/flows.js";
|
||||
import type { ExplorationFlow } from "../contracts/standalone-report.js";
|
||||
import { EXPLORATION_REPORT_KIND, type ExplorationFlow, type ExplorationReport } from "../contracts/standalone-report.js";
|
||||
import { compareFlowRuns, type FlowRun } from "../review/flow-comparison.js";
|
||||
import { diffScreenshots } from "../review/visual-evidence.js";
|
||||
import { agentDeviceLauncher, ensureController } from "../shared/controller-cache.js";
|
||||
import { CiError } from "../shared/ci-errors.js";
|
||||
|
||||
|
||||
/** `--alias NAME` plus a fixed token from EXPLORE_PREVIEW_TOKEN: the hosted link stays the same from one run to the next (validated by the deploy). */
|
||||
@@ -156,3 +160,32 @@ export async function exploreSession(): Promise<void> {
|
||||
openViewerInBrowser(viewer.url);
|
||||
await waitForViewerShutdown(viewer);
|
||||
}
|
||||
|
||||
/** A flows run's report: the report directory explore:binary or explore:session wrote, or the output directory holding it. */
|
||||
async function flowRun(path: string, name: "before" | "after"): Promise<FlowRun> {
|
||||
const refuse = (message: string) => { throw new CiError({ code: "FLOW_COMPARE_INPUT_INVALID", stage: "preflight", message,
|
||||
nextAction: `Pass --${name} the output directory of an explore:binary or explore:session run that used --flows (or its report directory).` }); };
|
||||
for (const directory of [resolve(path), resolve(path, "report")]) {
|
||||
let text: string;
|
||||
try { text = await readFile(join(directory, "exploration.json"), "utf8"); } catch { continue; }
|
||||
let report: ExplorationReport;
|
||||
try { report = JSON.parse(text) as ExplorationReport; } catch { return refuse(`${join(directory, "exploration.json")} is not JSON.`); }
|
||||
if (report?.kind !== EXPLORATION_REPORT_KIND) return refuse(`${join(directory, "exploration.json")} is not an exploration report.`);
|
||||
if (!Array.isArray(report.exploration?.flows)) return refuse(`The run in ${directory} did not follow a flows file, so it has no checkpoints to compare.`);
|
||||
return { report, directory };
|
||||
}
|
||||
return refuse(`No exploration.json was found in ${resolve(path)} or its report folder.`);
|
||||
}
|
||||
|
||||
/** `compare:flows --before DIR --after DIR [--output FILE]`: two runs of one flows file, checkpoint by checkpoint. */
|
||||
export async function compareFlows(): Promise<void> {
|
||||
const [before, after] = await Promise.all([flowRun(required("before"), "before"), flowRun(required("after"), "after")]);
|
||||
let command: string | undefined;
|
||||
const comparison = await compareFlowRuns(before, after, async (beforePath, afterPath) => {
|
||||
if (!command) { await ensureController("agent-device", { environment: process.env }); command = agentDeviceLauncher(process.env); }
|
||||
return diffScreenshots({ baselinePath: beforePath, currentPath: afterPath, command });
|
||||
});
|
||||
const path = flag("output");
|
||||
if (path) await writeFile(resolve(path), JSON.stringify(comparison, null, 2));
|
||||
output({ ok: true, ...(path ? { output: resolve(path) } : {}), comparison });
|
||||
}
|
||||
@@ -84,6 +84,8 @@ const CI = `CI (the installed workflows run these; they manage remote EAS Simula
|
||||
capture:baseline --project DIRECTORY --sha COMMIT --output TEMP_DIRECTORY [--config FILE] [--target ID | --platform ios|android] [--controller agent-device|argent] [--simulator eas|local] [--repack eas|local]
|
||||
source:verify --base-project DIRECTORY --head-project DIRECTORY
|
||||
--base-sha COMMIT --head-sha COMMIT --output DIRECTORY [--config FILE] [--scope all|changes] [--target ID | --platform ios|android]
|
||||
compare:flows --before DIRECTORY --after DIRECTORY [--output FILE]
|
||||
(two runs of one --flows file, checkpoint by checkpoint: unchanged, changed with where, or why not compared)
|
||||
compare --baseline DIRECTORY --current DIRECTORY --output FILE
|
||||
[--review-output FILE] [--catalog-output FILE] [--catalog-name NAME]
|
||||
[--base-surface-coverage FILE] [--head-surface-coverage FILE]
|
||||
|
||||
@@ -189,7 +189,7 @@ export const DUO_PANELS = {
|
||||
inner: { width: 951, height: 669, statusBar: { x: 883, y: 34, width: 40, height: 64 } },
|
||||
} as const;
|
||||
const DUO_STATUS_BAR_PADDING_POINTS = 4;
|
||||
const isFoldable = (platform: "ios" | "android" | undefined, device: string | undefined) => platform !== "android" && /\bduo\b/i.test(device ?? "");
|
||||
export const isFoldable = (platform: "ios" | "android" | undefined, device: string | undefined) => platform !== "android" && /\bduo\b/i.test(device ?? "");
|
||||
|
||||
/**
|
||||
* The areas the verdict leaves out for a capture: the platform's status-bar band and, on iOS, its home-indicator
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
import type { ExplorationReport } from "./standalone-report.js";
|
||||
|
||||
export const FLOW_COMPARISON_KIND = "explore.flow-comparison" as const;
|
||||
|
||||
/** One run's side of a checkpoint: its screenshot (a path inside that run's report directory), or why it has none. */
|
||||
export type CheckpointSide = { observationId: string; screenshot: string } | { missing: string };
|
||||
|
||||
/**
|
||||
* A checkpoint of one flow in two runs, paired by flow name and checkpoint name. `before` or `after` is absent when
|
||||
* that run's flows file has no such flow or checkpoint.
|
||||
*/
|
||||
export type CheckpointPair = { flow: string; checkpoint: string; before?: CheckpointSide; after?: CheckpointSide };
|
||||
|
||||
/** What a run was captured on: two runs that differ here can still be compared, but a difference may come from it. */
|
||||
export type RunConditions = { app: string; platform: string; device: string; osVersion?: string; locale?: string; orientation?: string; colorScheme?: string };
|
||||
export type RunConditionDifference = { field: keyof RunConditions; before: string; after: string };
|
||||
|
||||
export function runConditions(report: ExplorationReport): RunConditions {
|
||||
const environment = report.map.run.environment ?? {};
|
||||
return { app: report.exploration.app.bundleId, platform: report.exploration.app.platform, device: report.exploration.device.model,
|
||||
...(report.exploration.device.osVersion ? { osVersion: report.exploration.device.osVersion } : {}),
|
||||
...(environment.locale ? { locale: environment.locale } : {}), ...(environment.orientation ? { orientation: environment.orientation } : {}),
|
||||
...(environment.colorScheme ? { colorScheme: environment.colorScheme } : {}) };
|
||||
}
|
||||
|
||||
export function conditionDifferences(before: RunConditions, after: RunConditions): RunConditionDifference[] {
|
||||
const fields = ["app", "platform", "device", "osVersion", "locale", "orientation", "colorScheme"] as const;
|
||||
return fields.flatMap(field => (before[field] ?? "unknown") === (after[field] ?? "unknown") ? []
|
||||
: [{ field, before: before[field] ?? "unknown", after: after[field] ?? "unknown" }]);
|
||||
}
|
||||
|
||||
const keyOf = (name: string) => name.trim().toLowerCase();
|
||||
|
||||
function side(report: ExplorationReport, checkpoint: NonNullable<ExplorationReport["exploration"]["flows"]>[number]["checkpoints"][number]): CheckpointSide {
|
||||
if (checkpoint.observationId === undefined) return { missing: checkpoint.missing };
|
||||
const observation = report.map.observations.find(item => item.id === checkpoint.observationId);
|
||||
const artifact = observation?.screenshotArtifactId ? report.map.artifacts.find(item => item.id === observation.screenshotArtifactId) : undefined;
|
||||
const asset = artifact ? report.assets.find(item => item.sha256 === artifact.sha256) : undefined;
|
||||
return asset ? { observationId: checkpoint.observationId, screenshot: asset.path } : { missing: "The report has no screenshot for this checkpoint." };
|
||||
}
|
||||
|
||||
/**
|
||||
* Pairs the checkpoints of two runs of a flows file: the after run's flows and checkpoints in its order, then those only
|
||||
* the before run has. Names match as the flows file compares them, ignoring case and surrounding space.
|
||||
*/
|
||||
export function pairCheckpoints(before: ExplorationReport, after: ExplorationReport): CheckpointPair[] {
|
||||
const beforeFlows = before.exploration.flows ?? [], afterFlows = after.exploration.flows ?? [];
|
||||
const flowNames = [...afterFlows, ...beforeFlows.filter(flow => !afterFlows.some(other => keyOf(other.name) === keyOf(flow.name)))].map(flow => flow.name);
|
||||
return flowNames.flatMap(name => {
|
||||
const was = beforeFlows.find(flow => keyOf(flow.name) === keyOf(name)), now = afterFlows.find(flow => keyOf(flow.name) === keyOf(name));
|
||||
const wasPoints = was?.checkpoints ?? [], nowPoints = now?.checkpoints ?? [];
|
||||
const names = [...nowPoints, ...wasPoints.filter(point => !nowPoints.some(other => keyOf(other.name) === keyOf(point.name)))].map(point => point.name);
|
||||
return names.map(checkpoint => {
|
||||
const old = wasPoints.find(point => keyOf(point.name) === keyOf(checkpoint)), current = nowPoints.find(point => keyOf(point.name) === keyOf(checkpoint));
|
||||
return { flow: name, checkpoint, ...(old ? { before: side(before, old) } : {}), ...(current ? { after: side(after, current) } : {}) };
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
import { open } from "node:fs/promises";
|
||||
import { resolve } from "node:path";
|
||||
import { isFoldable, systemRegions } from "../contracts/capture.js";
|
||||
import { FLOW_COMPARISON_KIND, conditionDifferences, pairCheckpoints, runConditions, type CheckpointPair, type RunConditionDifference, type RunConditions } from "../contracts/flow-comparison.js";
|
||||
import type { ExplorationReport } from "../contracts/standalone-report.js";
|
||||
import { ignoreSystemRegions, systemBandFrameLoader } from "./evidence-assessment.js";
|
||||
import type { ScreenshotDiffMeasurement } from "./visual-evidence.js";
|
||||
|
||||
/**
|
||||
* `changed` and `unchanged` compare the two screenshots; `different-size` could not (another device or orientation);
|
||||
* `missing` has no screenshot on one side; `added` and `removed` are in only one run's flows file.
|
||||
*/
|
||||
export type CheckpointVerdict = "unchanged" | "changed" | "different-size" | "missing" | "added" | "removed";
|
||||
export type ComparedCheckpoint = CheckpointPair & {
|
||||
verdict: CheckpointVerdict;
|
||||
/** Share of the screenshot that differs once the status bar and home indicator are set aside. */
|
||||
mismatchPercentage?: number;
|
||||
/** Where the screenshots differ, in percent of the image (0–100). */
|
||||
regions?: Array<{ x: number; y: number; width: number; height: number }>;
|
||||
};
|
||||
|
||||
export type FlowComparison = {
|
||||
kind: typeof FLOW_COMPARISON_KIND;
|
||||
before: RunConditions & { report: string };
|
||||
after: RunConditions & { report: string };
|
||||
/** How the runs were captured differently; a checkpoint that changed may owe its change to these. */
|
||||
conditions: RunConditionDifference[];
|
||||
notes: string[];
|
||||
counts: Record<CheckpointVerdict, number>;
|
||||
checkpoints: ComparedCheckpoint[];
|
||||
};
|
||||
|
||||
export type FlowRun = { report: ExplorationReport; directory: string };
|
||||
export type MeasureScreenshots = (beforePath: string, afterPath: string) => Promise<ScreenshotDiffMeasurement>;
|
||||
|
||||
/** A PNG's pixel size, from its header. */
|
||||
async function pngSize(path: string): Promise<{ width: number; height: number }> {
|
||||
const file = await open(path, "r");
|
||||
try {
|
||||
const header = Buffer.alloc(24);
|
||||
await file.read(header, 0, 24, 0);
|
||||
return { width: header.readUInt32BE(16), height: header.readUInt32BE(20) };
|
||||
} finally { await file.close(); }
|
||||
}
|
||||
|
||||
/**
|
||||
* Compares two runs of the same flows file checkpoint by checkpoint, with the review's screenshot diff and the
|
||||
* status-bar and home-indicator regions it sets aside. `measure` is only called for pairs with a screenshot on both
|
||||
* sides of the same size.
|
||||
*/
|
||||
export async function compareFlowRuns(before: FlowRun, after: FlowRun, measure: MeasureScreenshots, threshold = 0.1): Promise<FlowComparison> {
|
||||
const was = runConditions(before.report), now = runConditions(after.report);
|
||||
const notes: string[] = [];
|
||||
// A Duo's status bar sits on whichever panel was in use, and a run does not record the pose.
|
||||
const foldable = isFoldable(before.report.exploration.app.platform, was.device);
|
||||
if (foldable) notes.push("The status bar is not set aside: the run does not record which panel of the foldable was in use.");
|
||||
const system = foldable ? [] : systemRegions({ platform: before.report.exploration.app.platform, device: was.device });
|
||||
const checkpoints: ComparedCheckpoint[] = [];
|
||||
for (const pair of pairCheckpoints(before.report, after.report)) {
|
||||
if (!pair.before) { checkpoints.push({ ...pair, verdict: "added" }); continue; }
|
||||
if (!pair.after) { checkpoints.push({ ...pair, verdict: "removed" }); continue; }
|
||||
if (!("screenshot" in pair.before) || !("screenshot" in pair.after)) { checkpoints.push({ ...pair, verdict: "missing" }); continue; }
|
||||
const beforePath = resolve(before.directory, pair.before.screenshot), afterPath = resolve(after.directory, pair.after.screenshot);
|
||||
const [a, b] = await Promise.all([pngSize(beforePath), pngSize(afterPath)]);
|
||||
if (a.width !== b.width || a.height !== b.height) { checkpoints.push({ ...pair, verdict: "different-size" }); continue; }
|
||||
const raw = await measure(beforePath, afterPath);
|
||||
const { measurement } = ignoreSystemRegions(raw, system, systemBandFrameLoader(raw, threshold));
|
||||
const different = measurement.regions.reduce((sum, region) => sum + region.differentPixels, 0);
|
||||
const mismatchPercentage = raw.totalPixels ? Math.round((different / raw.totalPixels) * 10000) / 100 : 0;
|
||||
// As in a review, a mismatch that rounds to 0.00% is rendering noise.
|
||||
const changed = measurement.regions.length > 0 && mismatchPercentage > 0;
|
||||
checkpoints.push({ ...pair, verdict: changed ? "changed" : "unchanged", mismatchPercentage,
|
||||
...(changed ? { regions: measurement.regions.map(region => region.normalizedRect) } : {}) });
|
||||
}
|
||||
const counts = { unchanged: 0, changed: 0, "different-size": 0, missing: 0, added: 0, removed: 0 };
|
||||
for (const item of checkpoints) counts[item.verdict] += 1;
|
||||
return { kind: FLOW_COMPARISON_KIND, before: { ...was, report: before.directory }, after: { ...now, report: after.directory },
|
||||
conditions: conditionDifferences(was, now), notes, counts, checkpoints };
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
import { createReadStream } from "node:fs";
|
||||
import { lstat, open } from "node:fs/promises";
|
||||
import { basename } from "node:path";
|
||||
import { createGunzip } from "node:zlib";
|
||||
import { Reader, Uint8ArrayWriter, ZipReader, configure } from "@zip.js/zip.js";
|
||||
import { CiError } from "./ci-errors.js";
|
||||
|
||||
/**
|
||||
* Reads the entries of tar.gz and zip archives without unpacking them: a prebuilt app, a downloaded controller, a
|
||||
* baseline artifact. Nothing here writes a file; callers place entries themselves, through safeRelativePath.
|
||||
*/
|
||||
|
||||
export const MAX_METADATA_BYTES = 4 * 1024 * 1024;
|
||||
export const MAX_ARCHIVE_ENTRIES = 200_000;
|
||||
|
||||
export function archiveError(code: string, message: string, nextAction: string, details?: Record<string, unknown>): CiError {
|
||||
return new CiError({ code, stage: "archive", message, nextAction, ...(details ? { details } : {}) });
|
||||
}
|
||||
|
||||
// ---- tar.gz -------------------------------------------------------------------------------------
|
||||
|
||||
type TarVisitor = (entry: { name: string; type: string; size: number; mode: number }, contents: () => Promise<Buffer>, skip: () => Promise<void>) => Promise<boolean | void>;
|
||||
|
||||
/** Walk a gzip tarball once; `visit` returns true to stop. Long GNU names are honoured; nothing is written. */
|
||||
export async function walkTar(path: string, visit: TarVisitor): Promise<void> {
|
||||
const stream = createReadStream(path).pipe(createGunzip());
|
||||
const iterator = stream[Symbol.asyncIterator]();
|
||||
let buffer = Buffer.alloc(0);
|
||||
let done = false;
|
||||
const fill = async (needed: number) => {
|
||||
while (buffer.length < needed && !done) {
|
||||
const next = await iterator.next();
|
||||
if (next.done) { done = true; break; }
|
||||
buffer = Buffer.concat([buffer, next.value as Buffer]);
|
||||
}
|
||||
};
|
||||
const take = async (count: number): Promise<Buffer> => {
|
||||
await fill(count);
|
||||
if (buffer.length < count) throw archiveError("ARCHIVE_INVALID", "The tarball ended unexpectedly.", "Pass a complete .tar.gz archive.");
|
||||
const out = buffer.subarray(0, count); buffer = buffer.subarray(count); return Buffer.from(out);
|
||||
};
|
||||
const drop = async (count: number) => {
|
||||
let remaining = count;
|
||||
while (remaining > 0) {
|
||||
await fill(Math.min(remaining, 1 << 20));
|
||||
const step = Math.min(remaining, buffer.length);
|
||||
if (!step) throw archiveError("ARCHIVE_INVALID", "The tarball ended unexpectedly.", "Pass a complete .tar.gz archive.");
|
||||
buffer = buffer.subarray(step); remaining -= step;
|
||||
}
|
||||
};
|
||||
let longName: string | undefined;
|
||||
let entries = 0;
|
||||
try {
|
||||
for (;;) {
|
||||
await fill(512);
|
||||
if (buffer.length < 512 || buffer.subarray(0, 512).every(byte => byte === 0)) return;
|
||||
const header = await take(512);
|
||||
const field = (offset: number, length: number) => header.subarray(offset, offset + length).toString("utf8").replace(/\0.*$/s, "");
|
||||
const size = parseInt(field(124, 12).trim() || "0", 8);
|
||||
const type = field(156, 1) || "0";
|
||||
const prefix = field(345, 155);
|
||||
const name = longName ?? (prefix ? `${prefix}/${field(0, 100)}` : field(0, 100));
|
||||
longName = undefined;
|
||||
const padded = Math.ceil(size / 512) * 512;
|
||||
if (type === "L") { longName = (await take(padded)).subarray(0, size).toString("utf8").replace(/\0+$/, ""); continue; }
|
||||
if (++entries > MAX_ARCHIVE_ENTRIES) throw archiveError("ARCHIVE_TOO_LARGE", "The tarball has too many entries.", "Pass a smaller archive.");
|
||||
let consumed = false;
|
||||
const stop = await visit({ name, type, size, mode: parseInt(field(100, 8).trim() || "0", 8) },
|
||||
async () => { consumed = true; return (await take(padded)).subarray(0, size); },
|
||||
async () => { consumed = true; await drop(padded); });
|
||||
if (!consumed) await drop(padded);
|
||||
if (stop) return;
|
||||
}
|
||||
} finally { await iterator.return?.(undefined); }
|
||||
}
|
||||
|
||||
export function safeRelativePath(name: string): string | undefined {
|
||||
const parts = name.replace(/^\.\//, "").split("/").filter(part => part.length);
|
||||
if (!parts.length || parts.some(part => part === "." || part === ".." || part.includes("\\") || part.includes("\0"))) return undefined;
|
||||
return parts.join("/");
|
||||
}
|
||||
|
||||
// ---- zip ----------------------------------------------------------------------------------------
|
||||
|
||||
export type ZipEntry = Awaited<ReturnType<ZipReader<unknown>["getEntries"]>>[number];
|
||||
|
||||
/** zip.js reads the central directory and entries by range, so a 200 MB APK is never loaded whole. */
|
||||
class FileRangeReader extends Reader<unknown> {
|
||||
constructor(private readonly path: string) { super(path); }
|
||||
override async init(): Promise<void> {
|
||||
this.size = (await lstat(this.path)).size;
|
||||
}
|
||||
/** Each range opens its own descriptor: nothing stays open after the entries are read, so no handle leaks when a caller stops early. */
|
||||
override async readUint8Array(index: number, length: number): Promise<Uint8Array> {
|
||||
const out = new Uint8Array(length);
|
||||
const handle = await open(this.path, "r");
|
||||
try {
|
||||
let offset = 0;
|
||||
while (offset < length) {
|
||||
const { bytesRead } = await handle.read(out, offset, length - offset, index + offset);
|
||||
if (bytesRead === 0) break;
|
||||
offset += bytesRead;
|
||||
}
|
||||
return offset === length ? out : out.subarray(0, offset);
|
||||
} finally { await handle.close(); }
|
||||
}
|
||||
}
|
||||
|
||||
const readers = new WeakMap<object, FileRangeReader>();
|
||||
|
||||
/** The ABIs an APK ships native libraries for (`lib/<abi>/`), without reading its manifest. */
|
||||
export async function apkAbis(path: string): Promise<string[]> {
|
||||
return [...new Set((await zipEntries(path)).map(entry => /^lib\/([^/]+)\//.exec(entry.filename)?.[1]).filter((abi): abi is string => !!abi))].sort();
|
||||
}
|
||||
|
||||
export async function zipEntries(path: string): Promise<ZipEntry[]> {
|
||||
configure({ useWebWorkers: false });
|
||||
const reader = new FileRangeReader(path);
|
||||
const zip = new ZipReader(reader, { useWebWorkers: false });
|
||||
try {
|
||||
const entries = await zip.getEntries();
|
||||
if (entries.length > MAX_ARCHIVE_ENTRIES) throw archiveError("ARCHIVE_TOO_LARGE", "The archive has too many entries.", "Pass a smaller archive.");
|
||||
for (const entry of entries) readers.set(entry, reader);
|
||||
return entries;
|
||||
} catch (error) {
|
||||
if (error instanceof CiError) throw error;
|
||||
throw archiveError("ARCHIVE_INVALID", `${basename(path)} is not a readable ZIP archive: ${error instanceof Error ? error.message : String(error)}`, "Pass a valid .ipa or .apk.");
|
||||
}
|
||||
}
|
||||
|
||||
export async function readZipEntry(entry: ZipEntry, maxBytes = MAX_METADATA_BYTES): Promise<Uint8Array> {
|
||||
if (entry.directory || !entry.getData) throw archiveError("ARCHIVE_INVALID", `${entry.filename} is not a file entry.`, "Pass a valid archive.");
|
||||
if (entry.uncompressedSize > maxBytes) throw archiveError("ARCHIVE_INVALID", `${entry.filename} is too large.`, "Pass a valid archive.");
|
||||
return entry.getData(new Uint8ArrayWriter(), { checkSignature: true, useWebWorkers: false });
|
||||
}
|
||||
@@ -11,7 +11,7 @@ import { chmod, mkdir, open, readdir, rename, rm, stat, writeFile } from "node:f
|
||||
import { homedir } from "node:os";
|
||||
import { dirname, join, resolve, sep } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { safeRelativePath, walkTar } from "../binary/archive.js";
|
||||
import { safeRelativePath, walkTar } from "./archive-entries.js";
|
||||
import { CiError } from "./ci-errors.js";
|
||||
import { AGENT_DEVICE_VERSION } from "./agent-device-version.js";
|
||||
import { ARGENT_VERSION } from "./argent-version.js";
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
// instance), and every tsImport call loads its own copy of each module.
|
||||
export { runBlackBoxExploration } from "../../src/binary/explorer.js";
|
||||
export { assertFlowsFit, createFlowSession, flowsNeedModel, parseFlowPlan } from "../../src/binary/flows.js";
|
||||
export { hintInputs } from "../../src/binary/run.js";
|
||||
export { chooseStrategy, startStrategy, strategyInputs } from "../../src/binary/strategy.js";
|
||||
export { runBoundedSession } from "../../src/capture/agent/session-shared.js";
|
||||
export { ExploreStore, validateMap } from "../../src/map/store.js";
|
||||
export { CiError } from "../../src/shared/ci-errors.js";
|
||||
|
||||
@@ -9,7 +9,8 @@ const { runBlackBoxExploration, screenEvidenceKey, elementsOf, fieldName, system
|
||||
const { ExploreStore, validateMap } = await tsImport("../../src/map/store.ts", import.meta.url);
|
||||
const { CONTRACT } = await tsImport("../../src/types.ts", import.meta.url);
|
||||
const tsRun = await tsImport("../../src/binary/run.ts", import.meta.url);
|
||||
const { assertAppUrl, describeForeground, foreignNode, guardPositionals, leftForeground, retryWhileBusy } = tsRun;
|
||||
const { describeForeground, foreignNode, guardPositionals, leftForeground, retryWhileBusy } = tsRun;
|
||||
const { assertAppUrl } = await tsImport("../../src/binary/app-url.ts", import.meta.url);
|
||||
|
||||
const PNG = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==", "base64");
|
||||
const SCREENS = {
|
||||
@@ -246,7 +247,8 @@ test("a {{secret:NAME}} from the hints is typed on the device, while the model,
|
||||
});
|
||||
|
||||
test("hint inputs are read before a session starts and name what to set", async () => {
|
||||
const { hintInputs } = await tsImport("../../src/binary/run.ts", import.meta.url);
|
||||
const { strategyInputs } = await tsImport("../../src/binary/strategy.ts", import.meta.url);
|
||||
const hintInputs = (hints, environment) => strategyInputs({ kind: "model" }, hints, environment);
|
||||
assert.deepEqual(hintInputs("Explore the settings", {}).names, []);
|
||||
assert.deepEqual(hintInputs("Use {{secret:APP_TEST_EMAIL}} and {{secret:APP_TEST_PASSWORD}}", { APP_TEST_EMAIL: "[email protected]", APP_TEST_PASSWORD: "pw" }).names, ["APP_TEST_EMAIL", "APP_TEST_PASSWORD"]);
|
||||
assert.throws(() => hintInputs("Use {{secret:APP_TEST_PASSWORD}}", {}), error => error.code === "EXPLORATION_SECRET_MISSING" && /export APP_TEST_PASSWORD=/.test(error.nextAction));
|
||||
@@ -854,7 +856,7 @@ test("a coordinate tap with no description on an icon-font button is named by wh
|
||||
});
|
||||
|
||||
test("with no --provider, model access falls through API keys and local logins to the first that passes preflight", async () => {
|
||||
const { preflightModelSession } = await tsImport("../../src/binary/run.ts", import.meta.url);
|
||||
const { preflightModelSession } = await tsImport("../../src/binary/strategy.ts", import.meta.url);
|
||||
const tried = [];
|
||||
const working = "anthropic/local-login";
|
||||
const createSession = (config, { auth }) => ({ async preflight() {
|
||||
@@ -870,7 +872,7 @@ test("with no --provider, model access falls through API keys and local logins t
|
||||
});
|
||||
|
||||
test("an explicit --provider is the only model access tried, and its preflight error is the one reported", async () => {
|
||||
const { preflightModelSession } = await tsImport("../../src/binary/run.ts", import.meta.url);
|
||||
const { preflightModelSession } = await tsImport("../../src/binary/strategy.ts", import.meta.url);
|
||||
const tried = [];
|
||||
const createSession = (config, { auth }) => ({ async preflight() { tried.push(`${config.provider}/${auth}`); throw new Error("refused"); } });
|
||||
await assert.rejects(preflightModelSession({ provider: "openai", createSession }, { ANTHROPIC_API_KEY: "key" }, () => {}), /refused/);
|
||||
@@ -878,7 +880,7 @@ test("an explicit --provider is the only model access tried, and its preflight e
|
||||
});
|
||||
|
||||
test("when nothing passes preflight, the first failure is reported", async () => {
|
||||
const { preflightModelSession } = await tsImport("../../src/binary/run.ts", import.meta.url);
|
||||
const { preflightModelSession } = await tsImport("../../src/binary/strategy.ts", import.meta.url);
|
||||
const createSession = (config, { auth }) => ({ async preflight() { throw new Error(`${config.provider} ${auth} refused`); } });
|
||||
await assert.rejects(preflightModelSession({ createSession }, {}, () => {}), /anthropic local-login refused/);
|
||||
});
|
||||
|
||||
@@ -5,7 +5,7 @@ import { join } from "node:path";
|
||||
import test from "node:test";
|
||||
import { tsImport } from "tsx/esm/api";
|
||||
|
||||
const { runBlackBoxExploration, assertFlowsFit, createFlowSession, flowsNeedModel, parseFlowPlan, hintInputs, runBoundedSession, ExploreStore, validateMap, CiError, CONTRACT } =
|
||||
const { runBlackBoxExploration, assertFlowsFit, createFlowSession, flowsNeedModel, parseFlowPlan, chooseStrategy, startStrategy, strategyInputs, runBoundedSession, ExploreStore, validateMap, CiError, CONTRACT } =
|
||||
await tsImport("../helpers/flows-host.ts", import.meta.url);
|
||||
|
||||
const PNG = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==", "base64");
|
||||
@@ -185,8 +185,21 @@ test("a flows file is checked before any device starts", () => {
|
||||
|
||||
test("a {{secret:NAME}} in a described step must be set before any device starts, as one in the hints must", () => {
|
||||
const plan = parseFlowPlan({ flows: [{ name: "Sign in", steps: [{ do: "Sign in with {{secret:TEST_PASSWORD}}" }, { checkpoint: "Home" }] }] });
|
||||
assert.throws(() => hintInputs(undefined, {}, undefined, plan), error => error.code === "EXPLORATION_SECRET_MISSING" && /TEST_PASSWORD/.test(error.nextAction));
|
||||
assert.deepEqual(hintInputs(undefined, { TEST_PASSWORD: "hunter2" }, undefined, plan).names, ["TEST_PASSWORD"]);
|
||||
assert.throws(() => strategyInputs({ kind: "flows", plan }, undefined, {}), error => error.code === "EXPLORATION_SECRET_MISSING" && /TEST_PASSWORD/.test(error.nextAction));
|
||||
assert.deepEqual(strategyInputs({ kind: "flows", plan }, undefined, { TEST_PASSWORD: "hunter2" }).names, ["TEST_PASSWORD"]);
|
||||
});
|
||||
|
||||
test("the strategy is chosen before any device starts, and options that would do nothing with it are refused", () => {
|
||||
const exact = { flows: [{ name: "A", steps: [{ action: "tap", label: "Settings" }, { checkpoint: "Settings" }] }] };
|
||||
const described = { flows: [{ name: "A", steps: [{ do: "Open the settings" }, { checkpoint: "Settings" }] }] };
|
||||
assert.deepEqual(chooseStrategy({}), { kind: "model" });
|
||||
assert.equal(chooseStrategy({ flows: exact }).kind, "flows");
|
||||
assert.equal(chooseStrategy({ flows: described, provider: "openai", hints: "Use the test account" }).kind, "flows");
|
||||
assert.throws(() => chooseStrategy({ flows: exact, replay: { steps: [{ capture: "Home" }] } }), { code: "FLOWS_WITH_REPLAY" });
|
||||
assert.throws(() => chooseStrategy({ flows: exact, model: "gpt-6.1-sol" }), error => error.code === "FLOWS_OPTIONS_IGNORED" && /--model would do nothing/.test(error.message));
|
||||
assert.throws(() => chooseStrategy({ replay: { steps: [{ capture: "Home" }] }, hints: "x" }), { code: "REPLAY_OPTIONS_IGNORED" });
|
||||
assert.throws(() => chooseStrategy({ flows: { flows: [{ name: "A", steps: [{ action: "open", url: "https://example.com" }, { checkpoint: "Web" }] }] } }),
|
||||
error => error.code === "FLOWS_PLAN_INVALID" && /Flow 1 \(A\), step 1: https: URLs leave the app/.test(error.message));
|
||||
});
|
||||
|
||||
test("a checkpoint after a relaunch takes a new picture, even of the screen the previous flow ended on untouched", async () => {
|
||||
@@ -264,8 +277,8 @@ test("a described step limited to another platform needs no model, and its secre
|
||||
assert.equal(flowsNeedModel(plan, "ios"), false);
|
||||
assert.equal(flowsNeedModel(plan, "android"), true);
|
||||
assert.doesNotThrow(() => createFlowSession(plan, "ios", { app: { name: "Example" } }));
|
||||
assert.deepEqual(hintInputs(undefined, {}, undefined, plan, "ios").names, []);
|
||||
assert.throws(() => hintInputs(undefined, {}, undefined, plan), error => error.code === "EXPLORATION_SECRET_MISSING", "without a platform, every step's secrets are needed");
|
||||
assert.deepEqual(strategyInputs({ kind: "flows", plan }, undefined, {}, "ios").names, []);
|
||||
assert.throws(() => strategyInputs({ kind: "flows", plan }, undefined, {}), error => error.code === "EXPLORATION_SECRET_MISSING", "without a platform, every step's secrets are needed");
|
||||
});
|
||||
|
||||
test("flows whose tool calls cannot fit the turn budget are refused before any device starts", () => {
|
||||
@@ -275,6 +288,14 @@ test("flows whose tool calls cannot fit the turn budget are refused before any d
|
||||
assert.doesNotThrow(() => assertFlowsFit(flows(41), "ios", limits), "41 flows take 124 calls");
|
||||
});
|
||||
|
||||
test("flows whose described steps run only on another platform start with no model at all", async () => {
|
||||
const plan = parseFlowPlan({ flows: [{ name: "A", steps: [{ do: "Sign in", platforms: ["android"] }, { action: "tap", label: "Settings" }, { checkpoint: "Settings" }] }] });
|
||||
const createSession = () => { throw new Error("no model may be started"); };
|
||||
const strategy = await startStrategy({ kind: "flows", plan }, { platform: "ios", limits: { maxScreens: 10, maxActions: 10 }, options: { createSession }, environment: {}, say() {}, app: { name: "Example" } });
|
||||
assert.deepEqual([strategy.provider, strategy.model, strategy.auth], ["flows", "flows file", "none"]);
|
||||
assert.deepEqual(strategy.flows(), [{ flow: "A", checkpoints: ["Settings"] }]);
|
||||
});
|
||||
|
||||
test("a checkpoint whose screen withholds its picture is recorded as missing, its flow goes on, and the summary names it", async () => {
|
||||
const plan = parseFlowPlan({ flows: [{ name: "Home", steps: [{ checkpoint: "Home" }, { action: "wait", milliseconds: 1 }, { checkpoint: "Still home" }, { action: "tap", label: "Settings" }, { checkpoint: "Settings" }] }] });
|
||||
const { result, map } = await explore(plan, { device: { withholdPicture: [2] } });
|
||||
@@ -330,7 +351,7 @@ test("a flow with no checkpoint on the platform is skipped, and its steps need n
|
||||
{ name: "Settings", steps: [{ action: "tap", label: "Settings" }, { checkpoint: "Settings" }] },
|
||||
] });
|
||||
assert.equal(flowsNeedModel(plan, "ios"), false);
|
||||
assert.deepEqual(hintInputs(undefined, {}, undefined, plan, "ios").names, []);
|
||||
assert.deepEqual(strategyInputs({ kind: "flows", plan }, undefined, {}, "ios").names, []);
|
||||
const { result } = await explore(plan);
|
||||
assert.deepEqual(result.flows.map(flow => flow.name), ["Settings"]);
|
||||
assert.match(result.summary, /^Ran 1 flow on ios/);
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdir, mkdtemp, writeFile } from "node:fs/promises";
|
||||
import { createRequire } from "node:module";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import test from "node:test";
|
||||
import { tsImport } from "tsx/esm/api";
|
||||
|
||||
const { pairCheckpoints, conditionDifferences, runConditions } = await tsImport("../../src/contracts/flow-comparison.ts", import.meta.url);
|
||||
const { compareFlowRuns } = await tsImport("../../src/review/flow-comparison.ts", import.meta.url);
|
||||
const { PNG } = createRequire(import.meta.url)("pngjs");
|
||||
|
||||
const png = (width, height) => PNG.sync.write(new PNG({ width, height }));
|
||||
|
||||
/**
|
||||
* A minimal exploration report of a flows run, written to a directory. `flows` maps each flow name to its checkpoints,
|
||||
* each `[name, size]` for a picture of that size or `[name, { missing }]` for one without.
|
||||
*/
|
||||
async function run(flows, options = {}) {
|
||||
const directory = await mkdtemp(join(tmpdir(), "explore-flow-compare-"));
|
||||
await mkdir(join(directory, "assets"));
|
||||
const report = { kind: "explore.exploration-report", assets: [],
|
||||
map: { run: { environment: { deviceModel: options.device ?? "iPhone 17 Pro", locale: options.locale ?? "en_US", orientation: "portrait", colorScheme: options.colorScheme ?? "light" } }, observations: [], artifacts: [] },
|
||||
exploration: { app: { name: "Example", bundleId: options.bundleId ?? "dev.example.app", platform: options.platform ?? "ios" }, device: { model: options.device ?? "iPhone 17 Pro", osVersion: options.osVersion ?? "26.0" }, flows: [] } };
|
||||
let index = 0;
|
||||
for (const [name, checkpoints] of Object.entries(flows)) {
|
||||
const flow = { name, status: "completed", checkpoints: [] };
|
||||
for (const [checkpoint, shot] of checkpoints) {
|
||||
if (shot.missing) { flow.checkpoints.push({ name: checkpoint, missing: shot.missing }); continue; }
|
||||
const sha = String(++index).padStart(64, "0"), id = `obs_${index}`;
|
||||
await writeFile(join(directory, "assets", `${sha}.png`), png(...shot));
|
||||
report.assets.push({ sha256: sha, path: `assets/${sha}.png` });
|
||||
report.map.artifacts.push({ id: `art_${index}`, kind: "screenshot", sha256: sha });
|
||||
report.map.observations.push({ id, screenshotArtifactId: `art_${index}` });
|
||||
flow.checkpoints.push({ name: checkpoint, observationId: id });
|
||||
}
|
||||
report.exploration.flows.push(flow);
|
||||
}
|
||||
return { report, directory };
|
||||
}
|
||||
|
||||
/** A diff that finds the given regions (in percent), each 1% of the pixels. */
|
||||
const measuring = (regionsFor = () => []) => async (baselinePath, currentPath) => {
|
||||
const regions = regionsFor(baselinePath, currentPath).map((normalizedRect, index) => ({ index, normalizedRect, rect: normalizedRect, differentPixels: 100, shareOfDiffPercentage: 0, densityPercentage: 0, shape: "", size: "" }));
|
||||
return { baselinePath, currentPath, totalPixels: 10_000, differentPixels: regions.length * 100, mismatchPercentage: regions.length, match: regions.length === 0, regions };
|
||||
};
|
||||
|
||||
test("checkpoints pair by flow and checkpoint name, ignoring case, in the after run's order", async () => {
|
||||
const before = await run({ "Sign in": [["Welcome", [4, 8]], ["Home", [4, 8]]], Settings: [["Settings", [4, 8]]] });
|
||||
const after = await run({ "sign in": [["home", [4, 8]], ["welcome", { missing: "The screenshot was blocked." }]], Profile: [["Profile", [4, 8]]] });
|
||||
const pairs = pairCheckpoints(before.report, after.report);
|
||||
assert.deepEqual(pairs.map(pair => [pair.flow, pair.checkpoint, !!pair.before, !!pair.after]), [
|
||||
["sign in", "home", true, true], ["sign in", "welcome", true, true], ["Profile", "Profile", false, true], ["Settings", "Settings", true, false]]);
|
||||
assert.deepEqual(pairs[1].after, { missing: "The screenshot was blocked." });
|
||||
assert.match(pairs[0].before.screenshot, /^assets\/0+2\.png$/);
|
||||
});
|
||||
|
||||
test("a run's conditions that differ are listed, not refused", async () => {
|
||||
const before = await run({ A: [["Home", [4, 8]]] }), after = await run({ A: [["Home", [4, 8]]] }, { osVersion: "27.0", colorScheme: "dark" });
|
||||
assert.deepEqual(conditionDifferences(runConditions(before.report), runConditions(after.report)),
|
||||
[{ field: "osVersion", before: "26.0", after: "27.0" }, { field: "colorScheme", before: "light", after: "dark" }]);
|
||||
});
|
||||
|
||||
test("each pair gets a verdict: unchanged, changed with where, missing, different-size, added or removed", async () => {
|
||||
const before = await run({ Flow: [["Same", [4, 8]], ["Moved", [4, 8]], ["Gone", [4, 8]], ["Blocked", [4, 8]], ["Rotated", [4, 8]]] });
|
||||
const after = await run({ Flow: [["Same", [4, 8]], ["Moved", [4, 8]], ["New", [4, 8]], ["Blocked", { missing: "A secret was on screen." }], ["Rotated", [8, 4]]] });
|
||||
const measured = [];
|
||||
const result = await compareFlowRuns(before, after, measuring((baseline, current) => {
|
||||
measured.push([baseline, current]);
|
||||
return baseline.endsWith(before.report.assets[1].path) ? [{ x: 10, y: 40, width: 20, height: 10 }] : [];
|
||||
}));
|
||||
assert.deepEqual(result.checkpoints.map(item => [item.checkpoint, item.verdict]),
|
||||
[["Same", "unchanged"], ["Moved", "changed"], ["New", "added"], ["Blocked", "missing"], ["Rotated", "different-size"], ["Gone", "removed"]]);
|
||||
assert.equal(measured.length, 2, "only pairs with a picture of the same size on both sides are diffed");
|
||||
assert.deepEqual(result.checkpoints[1].regions, [{ x: 10, y: 40, width: 20, height: 10 }]);
|
||||
assert.equal(result.checkpoints[1].mismatchPercentage, 1);
|
||||
assert.deepEqual(result.counts, { unchanged: 1, changed: 1, "different-size": 1, missing: 1, added: 1, removed: 1 });
|
||||
assert.deepEqual(result.conditions, []);
|
||||
});
|
||||
|
||||
test("a difference only in the status bar is set aside, as in a review", async () => {
|
||||
const before = await run({ Flow: [["Home", [4, 8]]] }), after = await run({ Flow: [["Home", [4, 8]]] });
|
||||
const result = await compareFlowRuns(before, after, measuring(() => [{ x: 80, y: 1, width: 10, height: 2 }]));
|
||||
assert.equal(result.checkpoints[0].verdict, "unchanged");
|
||||
});
|
||||
|
||||
test("on a foldable, the status bar is not set aside and the result says why", async () => {
|
||||
const before = await run({ Flow: [["Home", [4, 8]]] }, { device: "iPhone Duo" }), after = await run({ Flow: [["Home", [4, 8]]] }, { device: "iPhone Duo" });
|
||||
// Where the folded panel's status bar is: guessing that panel would hide a change the open panel shows there.
|
||||
const result = await compareFlowRuns(before, after, measuring(() => [{ x: 86, y: 13, width: 4, height: 4 }]));
|
||||
assert.equal(result.checkpoints[0].verdict, "changed");
|
||||
assert.match(result.notes[0], /foldable/);
|
||||
});
|
||||
|
||||
test("compare:flows reads two output directories, needs no diff when no pair has two pictures, and refuses a run without flows", async () => {
|
||||
const { spawnSync } = await import("node:child_process");
|
||||
const root = join(import.meta.dirname, "../..");
|
||||
const cli = join(root, "dist", "cli.js");
|
||||
const before = await run({ Flow: [["Home", { missing: "Not reached." }]] }), after = await run({ Flow: [["Home", [4, 8]]] });
|
||||
// explore:binary leaves the report in <output>/report.
|
||||
const output = await mkdtemp(join(tmpdir(), "explore-flow-compare-out-"));
|
||||
await mkdir(join(output, "report", "assets"), { recursive: true });
|
||||
await writeFile(join(output, "report", "exploration.json"), JSON.stringify(after.report));
|
||||
await writeFile(join(before.directory, "exploration.json"), JSON.stringify(before.report));
|
||||
const compared = spawnSync(process.execPath, [cli, "compare:flows", "--before", before.directory, "--after", output], { cwd: root, encoding: "utf8" });
|
||||
assert.equal(compared.status, 0, compared.stderr);
|
||||
const result = JSON.parse(compared.stdout);
|
||||
assert.deepEqual(result.comparison.checkpoints.map(item => [item.checkpoint, item.verdict]), [["Home", "missing"]]);
|
||||
const plain = await run({});
|
||||
delete plain.report.exploration.flows;
|
||||
await writeFile(join(plain.directory, "exploration.json"), JSON.stringify(plain.report));
|
||||
const refused = spawnSync(process.execPath, [cli, "compare:flows", "--before", plain.directory, "--after", output], { cwd: root, encoding: "utf8" });
|
||||
assert.notEqual(refused.status, 0);
|
||||
assert.match(JSON.parse(refused.stderr).error, /did not follow a flows file/);
|
||||
});
|
||||
Reference in new issue
Block a user