Compare commits

..
Author SHA1 Message Date
Brent Vatne b48a228e8e Viewer: a Flows view for compare:flows --report, each flow a filmstrip of its checkpoints
compare:flows --report DIR writes a report that opens on the Viewer's new Flows view: the after
run's report with the comparison beside it (exploration.json flowReview) and both builds'
checkpoint screenshots. Each checkpoint carries its verdict, changed regions, and the text its
screen showed (labels and values from the run's accessibility artifacts).

A flows run now records each flow's steps (actions, described steps, checkpoints) and the step
it failed at, so the review can show the steps that led to a checkpoint.

The Flows view lists the flows with what happened to each (Changed / All, Changed keeping every
run that stopped), and shows one flow as a filmstrip of its checkpoints, opened at its first
problem. Above the middle checkpoint are the steps that led to it, its name and status, and,
for a change too small to read in the picture, the text that changed, or why the run stopped.
Before | After (X) flips every picture in place; the arrows step the checkpoints, J and K the
changed areas. Nothing is saved.

The filmstrip is Screens' own, moved to components/shared/filmstrip.tsx; Screens wraps it.
Flows takes P and Ctrl+4, and appears only in a report with a review.
2026-10-24 18:05:44 -07:00
Brent Vatne d6b97331b3 Flows: compare two runs of a flows file checkpoint by checkpoint (compare:flows)
compare:flows --before DIR --after DIR [--output FILE] pairs the checkpoints of two runs by
flow and checkpoint name and diffs each pair with the review's screenshot diff, setting aside
the status bar and home indicator as a review does. Each checkpoint is unchanged, changed
(with the regions that differ), missing (no picture on one side, with why), different-size
(another device or orientation, so not diffed), added or removed.

The runs' conditions that differ (app, platform, device, OS version, locale, orientation,
appearance) are listed rather than refused, so an iPhone run and an Android run still line
up. On a foldable the status bar is not set aside, because a run does not record which panel
was in use; the result says so.

The pairing is a pure function in contracts/flow-comparison.ts; the diff is in
review/flow-comparison.ts. cli now imports review, which ARCHITECTURE lists.
2026-10-24 14:00:39 -07:00
Brent Vatne 6394778533 Merge the second round of flows fixes from #74: strategy.ts session inputs wait for the session's platform on a flows run
runSessionExploration now chooses its strategy before it knows the platform, so the #74 change
is ported onto strategyInputs: a flows run without --platform reads its secrets once the
session reports its platform.
2026-10-24 11:59:55 -07:00
Brent Vatne 66393ddbc9 Merge the flows fixes from #74: strategy.ts filters secrets and model access by platform, counts turns, and hands the explorer the checkpoints a flows run never attempted 2026-10-23 23:01:36 -07:00
Brent Vatne da719b0c31 Archives: the tar.gz and zip readers move to shared, and ARCHITECTURE describes capture strategies
shared/controller-cache, capture/local-simulator and ci/baseline-wait imported them from
binary/archive, a layer above them. ARCHITECTURE now lists binary/ as a layer, and a new section
says what a strategy decides and what the host (route-capture in a review, the explorer in an
exploration) owns.
2026-10-23 22:03:29 -07:00
Brent Vatne 1b0a0a1d96 Explorations: one place chooses and starts the strategy that drives the explorer
binary/strategy.ts holds what explore:binary and explore:session each branched on: chooseStrategy
checks the options before any device starts (a model, a replay plan or a flows file),
strategyInputs reads the {{secret:NAME}} inputs they need, and startStrategy returns the session
that drives the explorer's tools, with how the report names it. A new strategy is one more case
there. assertAppUrl moves to binary/app-url.ts so strategy.ts does not import run.ts.
2026-10-23 22:03:29 -07:00
42 changed files with 1950 additions and 606 deletions

No files matched your search

+14 -3
View File
@@ -29,10 +29,11 @@ This file describes the source layout and the rules that keep it in order.
| `review` | Compares base and head evidence, and prepares the local report assets. `ci` calls `evidence-assessment.ts` and `capture-review.ts`. | `evidence-assessment.ts`, `capture-review.ts`, `visual-evidence.ts`, `runtime-evidence.ts`, `evidence-comparator.ts`, `reviewer-changes.ts` |
| `report` | The report artifact, the pull-request comment, and the Viewer links. | `artifact.ts`, `pr-comment*.ts`, `viewer-links.ts` |
| `ci` | The pipeline behind `source:verify`, `capture:review` and `capture:baseline`. | `pipeline.ts`, `verified-plan.ts`, `artifact-cache.ts` |
| `binary` | Explores an app with no source (`explore:binary`, `explore:session`): the archive, the device session, the explorer, and the strategies that drive it. | `run.ts` (entry), `explorer.ts`, `strategy.ts`, `replay.ts`, `flows.ts`, `archive.ts`, `report.ts` |
| `cli` | Argument parsing, help, and the command handlers. | `args.ts`, `help.ts`, `*-commands.ts` |
| `map` | The local map store, build catalogs and comparisons, and the server for `expo-explore view`. | `store.ts`, `build-workspace.ts`, `compare.ts`, `viewer-server.ts` |
| `contracts` | The modules that `viewer/src` imports from `src/`, by value or by type: data shapes, validators and projections that the CLI and the Viewer share. `capture.ts` also defines the review types that the Viewer reads, such as `PublicReview`, and `review-validation.ts` validates them. | `capture.ts`, `review.ts`, `review-validation.ts`, `source-validation.ts` |
| `shared` | Small Node utilities that several folders use, and the pinned `agent-device` version that capture and review both check. | `ci-errors.ts`, `canonical-json.ts`, `bounded-bytes.ts`, `png-evidence.ts`, `agent-device-version.ts`, `timings.ts` (a capture job's timings and its summary, CI-FLOW §4.0) |
| `shared` | Small Node utilities that several folders use, and the pinned `agent-device` version that capture and review both check. | `ci-errors.ts`, `canonical-json.ts`, `bounded-bytes.ts`, `png-evidence.ts`, `archive-entries.ts` (tar.gz and zip readers), `agent-device-version.ts`, `timings.ts` (a capture job's timings and its summary, CI-FLOW §4.0) |
## Dependency rule
@@ -44,14 +45,24 @@ Each layer imports only from the layers below it.
3. `source` imports `source/route-providers`.
4. `capture` imports `capture/agent`, `config` and `map`. `capture/agent` imports `capture` and `config`.
5. `review` imports `capture` and `map`.
6. `ci` imports `config`, `source`, `capture`, `capture/agent`, `review` and `report`.
7. `cli` imports `ci`, `map`, `config` and `report`. `src/cli.ts` imports `cli`, `map` and `shared`.
6. `ci` imports `config`, `source`, `capture`, `capture/agent`, `review` and `report`. `binary` imports `config`, `capture`, `capture/agent`, `map` and `report`.
7. `cli` imports `ci`, `binary`, `map`, `config`, `report` and `review`. `src/cli.ts` imports `cli`, `map` and `shared`.
Every folder may import `contracts/`, `shared/` and `src/types.ts`.
The exception is `contracts/`: its value imports stay inside `contracts/` or go to `src/types.ts`.
The only cycle is `capture` and `capture/agent`, which are one feature.
Do not add a new import that points up a layer.
## Capture strategies
A capture decides three things: what to look at, how to get there, and what to record.
- **What to look at.** For a review, `source` plans the routes from the app's source (`capture-plan.ts`). An exploration has no source; its strategy decides as it goes.
- **How to get there.** In a review, `capture/route-capture.ts` opens each planned route by deep link, or, in agent mode, a model navigates to it (`capture/agent/runtime.ts`). In an exploration, a strategy drives the explorer's tools: a model exploring on its own, a replay plan (`binary/replay.ts`), or named flows (`binary/flows.ts`).
- **What to record.** The host records the evidence, never the strategy: `capture/route-capture.ts` in a review, `binary/explorer.ts` in an exploration. The host also owns the device, the budgets, refusing destructive and app-leaving actions, and redacting secrets. Both write an `ExploreMap` (`src/types.ts`, `schema/screen-map.v1.schema.json`). A review adds the capture bundle (`contracts/capture.ts`); an exploration adds the report in `contracts/standalone-report.ts`.
To add an exploration strategy, add a case to `StrategyChoice` in `binary/strategy.ts`: what it checks before any device starts, and the session it runs. The session is given the explorer's tools and calls them, as the model sessions, `createReplaySession` and `createFlowSession` do. Reviews have no such seam yet: the deep-link loop and agent mode are two branches of `capturePlannedRoutes`.
## The contracts boundary
`viewer/src` imports `src/` only through `src/contracts/` and `src/types.ts`.
+16
View File
@@ -267,6 +267,22 @@ npx @expo/explore explore:binary ./my-app.app --flows flows.json
- The report lists every flow and checkpoint of the file, in order, even those a run never reached: each one without a picture says why, so two runs still pair up step by step.
- Each flow is also a journey of the map: the path its steps took, from the screen it started on. When the app moved without a tap the map could record (it left the app, or changed on its own), the path goes on as a second journey of the flow.
- Relaunching ends the app's process, and the app keeps its data, so a flow that signs in leaves the next flow signed in. On `explore:session`, the app is relaunched before the first flow too.
To compare two runs of the same flows file, such as two builds, pass both output directories:
```sh
npx @expo/explore compare:flows --before explore-app-1 --after explore-app-2 --output flows.json
```
Each checkpoint comes back `unchanged`, `changed` (with where the screenshots differ, the status bar and home indicator set aside, as in a review), `missing` (no picture on one side, with why), `different-size` (another device or orientation, so not compared), or `added` / `removed` (in only one run's flows file). The result also lists how the two runs were captured differently (the app, device, OS version, locale, orientation, or appearance), since a change can come from those.
Add `--report DIRECTORY` to write a report that opens on the Viewer's Flows view, and host it with `report:upload DIRECTORY`. It is the after run's report with the comparison beside it and both builds' checkpoint screenshots:
- The list shows each flow with what happened to it ("Stopped · 2 not reached", "3 changed"). Changed keeps every flow with a change and every run that stopped.
- A flow is a filmstrip of its checkpoints, as in Screens, and opens at its first problem. Above the checkpoint in the middle are the steps that led to it, and, for a change too small to read in the picture, the text that changed. ← and → step through the checkpoints, J and K through the changed areas of one.
- Before | After (X) flips the pictures to the other build in place.
A run records each flow's steps for the review; runs made before Explore recorded them show none.
- A `{{secret:NAME}}` or a described step limited by `platforms` to another platform needs no value, and no model, on this one.
</details>
+1
View File
@@ -42,6 +42,7 @@ To compare runs, or an iPhone with an Android phone, screen for screen, pass `--
When the user describes a flow, or a set of flows, to look at ("sign in, then rename a pattern"), write them as a flows file and pass `--flows flows.json`, not `--replay`. The format is in the README ("Follow named flows, step by step"): `{ "flows": [{ "name": "Sign in", "steps": [{ "checkpoint": "Welcome" }, { "do": "Sign in with {{secret:TEST_EMAIL}}" }, { "checkpoint": "Home" }] }] }`.
- Each step is an exact action (the `--replay` format), a step described in words (`do`) that a model carries out, or a named `checkpoint`. Use exact actions where you know the labels, and `do` where you don't.
- Each flow starts from a relaunched app, and its checkpoints are photographed under their names. `report/exploration.json` lists each flow under `exploration.flows`, with its status and its checkpoints in order, so two runs pair up by flow and checkpoint name. A checkpoint on a screen that withholds its picture (a secret on it) is listed with why it has none, and its flow goes on.
- To compare two runs of the same flows file, run `npx @expo/explore compare:flows --before DIR --after DIR` with the two output directories. Each checkpoint is `unchanged`, `changed` (with regions), `missing`, `different-size`, `added` or `removed`, and `conditions` lists what differed between the runs (device, OS, locale, appearance). Add `--report DIR` to write a report that opens on the Viewer's Flows view (each flow's checkpoints, the steps between them and the text that changed), and host it with `report:upload DIR`.
- The file is checked before a device starts (`FLOWS_PLAN_INVALID`, `FLOWS_PLAN_EMPTY`, `FLOWS_PLAN_OVER_BUDGET`). `--provider`, `--model` and `--hints` steer the model that carries out `do` steps, and are refused when there are none (`FLOWS_OPTIONS_IGNORED`). `--flows` and `--replay` cannot be combined.
## Sign-in and other app state
+17
View File
@@ -0,0 +1,17 @@
import { CiError } from "../shared/ci-errors.js";
/** Schemes the OS routes to a browser, the store, Settings, or another app: opening one hands the session away from the app under test. */
const SYSTEM_URL_SCHEMES = /^(https?|ftp|file|javascript|data|blob|mailto|tel|telprompt|facetime(-audio)?|sms|mms|imessage|itms(-apps|-services|-beta)?|macappstore|app-settings|prefs|shortcuts|maps|music|videos|photos-redirect|x-apple-[a-z-]+|market|intent|content|android-app|geo|package):$/i;
const urlError = (code: string, message: string, nextAction: string) => new CiError({ code, stage: "session", message, nextAction });
/** Only the app's own URLs may be opened; an https page or a store link would hand the session to another app. */
export function assertAppUrl(url: string, appSchemes: readonly string[] = []): void {
let parsed: URL;
try { parsed = new URL(url); } catch { throw urlError("AGENT_URL_INVALID", `Not a URL: ${url}`, "Pass a custom-scheme app URL."); }
if (SYSTEM_URL_SCHEMES.test(parsed.protocol)) throw urlError("AGENT_URL_NOT_ALLOWED", `${parsed.protocol} URLs leave the app; only the app's own URL schemes are allowed.`, "Use the app's own URL scheme.");
const scheme = parsed.protocol.slice(0, -1).toLowerCase();
if (appSchemes.length && !appSchemes.includes(scheme)) {
throw urlError("AGENT_URL_NOT_ALLOWED", `${parsed.protocol} is not a URL scheme this app registers (${appSchemes.join(", ")}).`, "Use one of the app's own URL schemes.");
}
}
+1 -126
View File
@@ -2,9 +2,7 @@ import { createHash } from "node:crypto";
import { createReadStream } from "node:fs";
import { lstat, mkdir, open, readdir, readFile, realpath, writeFile, chmod } from "node:fs/promises";
import { basename, join, relative, resolve, sep } from "node:path";
import { createGunzip } from "node:zlib";
import { Reader, Uint8ArrayWriter, ZipReader, configure } from "@zip.js/zip.js";
import { CiError } from "../shared/ci-errors.js";
import { MAX_ARCHIVE_ENTRIES, MAX_METADATA_BYTES, archiveError, readZipEntry, safeRelativePath, walkTar, zipEntries } from "../shared/archive-entries.js";
import { APP_ICON_SOURCE_LIMIT, isWebp, shrinkAppIcon, webpSize } from "../capture/app-icon.js";
import { resolveResourceStrings } from "./android-resources.js";
import type { Platform } from "../types.js";
@@ -38,14 +36,8 @@ export type InspectedArchive = {
};
export const DEFAULT_MAX_ARCHIVE_BYTES = 1024 * 1024 * 1024;
const MAX_METADATA_BYTES = 4 * 1024 * 1024;
const MAX_ARCHIVE_ENTRIES = 200_000;
const MAX_RESOURCE_TABLE_BYTES = 32 * 1024 * 1024;
export function archiveError(code: string, message: string, nextAction: string, details?: Record<string, unknown>): CiError {
return new CiError({ code, stage: "archive", message, nextAction, ...(details ? { details } : {}) });
}
/** The archive kind from its name; the content check follows and may still reject it. */
export function archiveKind(path: string): ArchiveKind {
const name = basename(path).toLowerCase();
@@ -231,63 +223,6 @@ async function hashDirectory(root: string, maxBytes: number): Promise<{ sha256:
return { sha256: hash.digest("hex"), bytes };
}
// ---- tar.gz -------------------------------------------------------------------------------------
type TarVisitor = (entry: { name: string; type: string; size: number; mode: number }, contents: () => Promise<Buffer>, skip: () => Promise<void>) => Promise<boolean | void>;
/** Walk a gzip tarball once; `visit` returns true to stop. Long GNU names are honoured; nothing is written. */
export async function walkTar(path: string, visit: TarVisitor): Promise<void> {
const stream = createReadStream(path).pipe(createGunzip());
const iterator = stream[Symbol.asyncIterator]();
let buffer = Buffer.alloc(0);
let done = false;
const fill = async (needed: number) => {
while (buffer.length < needed && !done) {
const next = await iterator.next();
if (next.done) { done = true; break; }
buffer = Buffer.concat([buffer, next.value as Buffer]);
}
};
const take = async (count: number): Promise<Buffer> => {
await fill(count);
if (buffer.length < count) throw archiveError("ARCHIVE_INVALID", "The tarball ended unexpectedly.", "Pass a complete .tar.gz archive.");
const out = buffer.subarray(0, count); buffer = buffer.subarray(count); return Buffer.from(out);
};
const drop = async (count: number) => {
let remaining = count;
while (remaining > 0) {
await fill(Math.min(remaining, 1 << 20));
const step = Math.min(remaining, buffer.length);
if (!step) throw archiveError("ARCHIVE_INVALID", "The tarball ended unexpectedly.", "Pass a complete .tar.gz archive.");
buffer = buffer.subarray(step); remaining -= step;
}
};
let longName: string | undefined;
let entries = 0;
try {
for (;;) {
await fill(512);
if (buffer.length < 512 || buffer.subarray(0, 512).every(byte => byte === 0)) return;
const header = await take(512);
const field = (offset: number, length: number) => header.subarray(offset, offset + length).toString("utf8").replace(/\0.*$/s, "");
const size = parseInt(field(124, 12).trim() || "0", 8);
const type = field(156, 1) || "0";
const prefix = field(345, 155);
const name = longName ?? (prefix ? `${prefix}/${field(0, 100)}` : field(0, 100));
longName = undefined;
const padded = Math.ceil(size / 512) * 512;
if (type === "L") { longName = (await take(padded)).subarray(0, size).toString("utf8").replace(/\0+$/, ""); continue; }
if (++entries > MAX_ARCHIVE_ENTRIES) throw archiveError("ARCHIVE_TOO_LARGE", "The tarball has too many entries.", "Pass a smaller archive.");
let consumed = false;
const stop = await visit({ name, type, size, mode: parseInt(field(100, 8).trim() || "0", 8) },
async () => { consumed = true; return (await take(padded)).subarray(0, size); },
async () => { consumed = true; await drop(padded); });
if (!consumed) await drop(padded);
if (stop) return;
}
} finally { await iterator.return?.(undefined); }
}
async function tarEntry(path: string, match: (name: string) => boolean): Promise<{ name: string; contents: Buffer } | undefined> {
let found: { name: string; contents: Buffer } | undefined;
await walkTar(path, async (entry, contents) => {
@@ -299,12 +234,6 @@ async function tarEntry(path: string, match: (name: string) => boolean): Promise
return found;
}
export function safeRelativePath(name: string): string | undefined {
const parts = name.replace(/^\.\//, "").split("/").filter(part => part.length);
if (!parts.length || parts.some(part => part === "." || part === ".." || part.includes("\\") || part.includes("\0"))) return undefined;
return parts.join("/");
}
/**
* Unpack the iOS bundle out of a .tar.gz or simulator .ipa into `destination` for installation.
* Paths are confined to the destination; symbolic and hard links are rejected (an app bundle has none).
@@ -347,60 +276,6 @@ export async function extractIosBundle(archive: InspectedArchive, destination: s
return appDirectory;
}
// ---- zip ----------------------------------------------------------------------------------------
type ZipEntry = Awaited<ReturnType<ZipReader<unknown>["getEntries"]>>[number];
/** zip.js reads the central directory and entries by range, so a 200 MB APK is never loaded whole. */
class FileRangeReader extends Reader<unknown> {
constructor(private readonly path: string) { super(path); }
override async init(): Promise<void> {
this.size = (await lstat(this.path)).size;
}
/** Each range opens its own descriptor: nothing stays open after the entries are read, so no handle leaks when a caller stops early. */
override async readUint8Array(index: number, length: number): Promise<Uint8Array> {
const out = new Uint8Array(length);
const handle = await open(this.path, "r");
try {
let offset = 0;
while (offset < length) {
const { bytesRead } = await handle.read(out, offset, length - offset, index + offset);
if (bytesRead === 0) break;
offset += bytesRead;
}
return offset === length ? out : out.subarray(0, offset);
} finally { await handle.close(); }
}
}
const readers = new WeakMap<object, FileRangeReader>();
/** The ABIs an APK ships native libraries for (`lib/<abi>/`), without reading its manifest. */
export async function apkAbis(path: string): Promise<string[]> {
return [...new Set((await zipEntries(path)).map(entry => /^lib\/([^/]+)\//.exec(entry.filename)?.[1]).filter((abi): abi is string => !!abi))].sort();
}
async function zipEntries(path: string): Promise<ZipEntry[]> {
configure({ useWebWorkers: false });
const reader = new FileRangeReader(path);
const zip = new ZipReader(reader, { useWebWorkers: false });
try {
const entries = await zip.getEntries();
if (entries.length > MAX_ARCHIVE_ENTRIES) throw archiveError("ARCHIVE_TOO_LARGE", "The archive has too many entries.", "Pass a smaller archive.");
for (const entry of entries) readers.set(entry, reader);
return entries;
} catch (error) {
if (error instanceof CiError) throw error;
throw archiveError("ARCHIVE_INVALID", `${basename(path)} is not a readable ZIP archive: ${error instanceof Error ? error.message : String(error)}`, "Pass a valid .ipa or .apk.");
}
}
async function readZipEntry(entry: ZipEntry, maxBytes = MAX_METADATA_BYTES): Promise<Uint8Array> {
if (entry.directory || !entry.getData) throw archiveError("ARCHIVE_INVALID", `${entry.filename} is not a file entry.`, "Pass a valid archive.");
if (entry.uncompressedSize > maxBytes) throw archiveError("ARCHIVE_INVALID", `${entry.filename} is too large.`, "Pass a valid archive.");
return entry.getData(new Uint8ArrayWriter(), { checkSignature: true, useWebWorkers: false });
}
// ---- plist --------------------------------------------------------------------------------------
/** Property list reader for Info.plist: XML (Xcode default) or binary (`bplist00`). */
+11
View File
@@ -3,6 +3,7 @@ import { CiError } from "../shared/ci-errors.js";
import type { CiAgentSession, CiAgentSessionRunRequest, CiAgentSessionRunResult, CiAgentToolResult } from "../capture/agent/session.js";
import type { CiAgentTool } from "../capture/agent/provider.js";
import type { CaptureSetupAction } from "../config/explore-config.js";
import type { ExplorationFlowStep } from "../contracts/standalone-report.js";
import { explorationSecretNames } from "../config/exploration-inputs.js";
import { TOOL_LIMITS } from "./explorer.js";
import { actInput, describeAction, listed, parseReplayStep, type ReplayAction, type ReplayPlatform } from "./replay.js";
@@ -22,6 +23,8 @@ export type FlowSession = {
budgetNote(): string | undefined;
/** Each flow's checkpoints the run has not attempted, in order: the explorer records them as missing. */
notReached(): Array<{ flow: string; checkpoints: string[] }>;
/** Each flow's steps on the platform, and the one it failed at, for the report. */
steps(): Array<{ flow: string; steps: ExplorationFlowStep[]; failedStep?: number }>;
};
const MAX_FLOWS = 50;
@@ -228,11 +231,16 @@ export function createFlowSession(plan: FlowPlan, platform: ReplayPlatform, opti
const checkpoints = steps.flatMap(({ step }) => "checkpoint" in step && !attempted.get(flow.name)?.has(step.checkpoint) ? [step.checkpoint] : []);
return checkpoints.length ? [{ flow: flow.name, checkpoints }] : [];
});
/** The step each failed flow stopped at, counted from 1 among its steps on the platform. */
const failedAt = new Map<string, number>();
return {
provider: "flows",
model: "flows file",
async preflight() {},
notReached,
steps: () => flows.map(({ flow, steps }) => ({ flow: flow.name,
steps: steps.map(({ step }): ExplorationFlowStep => "checkpoint" in step ? { checkpoint: step.checkpoint } : "do" in step ? { do: step.do } : { action: describeAction(step) }),
...(failedAt.has(flow.name) ? { failedStep: failedAt.get(flow.name)! } : {}) })),
budgetNote() {
const names = notReached().flatMap(({ flow, checkpoints }) => checkpoints.map(name => `${flow} › ${name}`));
return names.length ? `Not reached: ${listed(names, TOOL_LIMITS.reason, "checkpoints")}.` : undefined;
@@ -273,6 +281,7 @@ export function createFlowSession(plan: FlowPlan, platform: ReplayPlatform, opti
continue;
}
failure = `Checkpoint "${step.checkpoint}" could not be taken: ${why(taken.value)}`;
failedAt.set(flow.name, index + 1);
break;
}
if ("do" in step) {
@@ -284,12 +293,14 @@ export function createFlowSession(plan: FlowPlan, platform: ReplayPlatform, opti
usage.inputTokens += goal.usage.inputTokens; usage.outputTokens += goal.usage.outputTokens;
if (goal.reached) continue;
failure = `Step ${number} ("${step.do.slice(0, 120)}") was not done: ${goal.reason}`;
failedAt.set(flow.name, index + 1);
break;
}
const acted = await call("act", actInput(step));
if (!acted.failed) continue;
if (step.optional) { skipped.push(`${flow.name} step ${number} (${describeAction(step)})`); continue; }
failure = `Step ${number} (${describeAction(step)}) failed: ${why(acted.value)}`;
failedAt.set(flow.name, index + 1);
break;
}
await call("end_flow", failure ? { status: "failed", reason: failure.slice(0, TOOL_LIMITS.reason) } : { status: "completed" });
+30 -162
View File
@@ -2,11 +2,12 @@ import { createHash, randomUUID } from "node:crypto";
import { lstat, mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createCiAgentSession, type CiAgentSession, type CiAgentSessionConfig } from "../capture/agent/session.js";
import { createLocalCiAgentSession } from "../capture/agent/session-local.js";
import type { CiAgentSession, CiAgentSessionConfig } from "../capture/agent/session.js";
import type { CiAgentProviderName } from "../capture/agent/provider.js";
import { assertReplayFits, createReplaySession, parseReplayPlan, replaySetup, type ReplayPlan } from "./replay.js";
import { assertFlowsFit, createFlowSession, flowGoals, flowSetup, flowsNeedModel, parseFlowPlan, type FlowPlan } from "./flows.js";
import type { ReplayPlan } from "./replay.js";
import type { FlowPlan } from "./flows.js";
import { assertAppUrl } from "./app-url.js";
import { chooseStrategy, startStrategy, strategyInputs, turnBudget, type ExplorationStrategy, type ModelAuth } from "./strategy.js";
import { CONTROLLER_SESSION, baseEnvironment, isRecord } from "../capture/common.js";
import { controllerData, defaultRunner, openedSimulatorDevice } from "../capture/controller.js";
import type { CommandRequest, CommandResult } from "../capture/run.js";
@@ -22,7 +23,7 @@ import { screenSpace } from "../capture/route-capture.js";
import { approveSessionUrlSchemes, simulatorControlActions, type EasCommand } from "../capture/simulator-control.js";
import { DEFAULT_ANDROID_DEVICE } from "../config/explore-config.js";
import { DUO_PANELS } from "../contracts/capture.js";
import { createExplorationInputs } from "../config/exploration-inputs.js";
import type { createExplorationInputs } from "../config/exploration-inputs.js";
import { ExploreStore, getCoverage } from "../map/store.js";
import { whoamiAccounts } from "../report/preview-deploy.js";
import { CiError, serializeCiError, toCiError } from "../shared/ci-errors.js";
@@ -34,7 +35,6 @@ import { runBlackBoxExploration, type BlackBoxExplorerResult, type BlackBoxFrame
export const DEFAULT_IOS_DEVICE = "iPhone 17 Pro";
export const DEFAULT_BINARY_PROJECT = "expo-explore";
export const DEFAULT_MODELS: Record<CiAgentProviderName, string> = { anthropic: "claude-sonnet-4-5", openai: "gpt-6.1-sol" };
export const BINARY_LIMITS = { maxDurationMinutes: 15, maxScreens: 40, maxActions: 150 } as const;
export type BinaryExploreOptions = {
@@ -76,7 +76,7 @@ export type BinaryExploreResult = {
archive: Omit<InspectedArchive, "path"> & { path: string };
session: { id: string; platform: Platform; device: string; owner: string; project: string };
runtime: ActiveSimulatorRuntime;
model: { provider: CiAgentProviderName | "replay" | "flows"; model: string; auth: "local-login" | "api-key" | "none" };
model: Pick<ExplorationStrategy, "provider" | "model" | "auth">;
exploration: BlackBoxExplorerResult;
durationMs: number;
};
@@ -166,23 +166,6 @@ export function resolveSessionApp(runner: typeof defaultRunner, connection: Simu
"Pass --bundle-id with the app's bundle identifier (iOS) or package name (Android).", { installed });
}
/**
* App-test inputs: each {{secret:NAME}} in the hints names an environment variable whose value Explore types for the
* model. Read before any session starts, so a missing one costs nothing.
*/
export function hintInputs(hints: string | undefined, environment: NodeJS.ProcessEnv, replay?: ReplayPlan, flows?: FlowPlan, platform?: Platform): ReturnType<typeof createExplorationInputs> {
const setup = replay ? replaySetup(replay) : flows ? flowSetup(flows, platform) : undefined;
try { return createExplorationInputs({ instructions: [hints ?? "", ...(flows ? flowGoals(flows, platform) : [])].join("\n"), ...(setup ? { setup } : {}), environment }); }
catch (error) {
if (!(error instanceof CiError) || !error.code.startsWith("EXPLORATION_SECRET_")) throw error;
const name = typeof error.details?.secretName === "string" ? error.details.secretName : "NAME";
throw new CiError({ code: error.code, stage: "preflight", message: error.message, ...(error.details ? { details: error.details } : {}),
nextAction: error.code === "EXPLORATION_SECRET_REFERENCE_INVALID"
? "Write each input in --hints as {{secret:NAME}} with an uppercase environment variable name that is not a provider, Expo, or CI credential."
: `Set ${name} in the environment that runs Explore (export ${name}=…) to the single-line test value, then rerun.` });
}
}
function validatedLimits(options: Pick<BinaryExploreOptions, "maxDurationMinutes" | "maxScreens" | "maxActions">) {
const limits = { maxDurationMinutes: options.maxDurationMinutes ?? BINARY_LIMITS.maxDurationMinutes, maxScreens: options.maxScreens ?? BINARY_LIMITS.maxScreens, maxActions: options.maxActions ?? BINARY_LIMITS.maxActions };
for (const [name, value] of Object.entries(limits)) if (!Number.isSafeInteger(value) || value < 1 || value > 1000) throw binaryError("EXPLORE_LIMIT_INVALID", `${name} must be an integer between 1 and 1000.`, "Correct the budget flag and rerun.");
@@ -193,108 +176,6 @@ function binaryError(code: string, message: string, nextAction: string, details?
return new CiError({ code, stage: "session", message, nextAction, ...(details ? { details } : {}) });
}
type ModelAuth = "local-login" | "api-key";
type ModelSessionOptions = Pick<BinaryExploreOptions, "provider" | "model" | "createSession">;
function modelSession(provider: CiAgentProviderName, auth: ModelAuth, options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
const config = { provider, model: options.model ?? DEFAULT_MODELS[provider] };
const session = options.createSession ? options.createSession(config, { env: environment, auth })
: auth === "api-key" ? createCiAgentSession(config, { env: environment }) : createLocalCiAgentSession(config, { env: environment });
return { session, provider, model: config.model, auth };
}
/** The turns the model gets: three for each action and two for each screen, plus a few to look around and finish. */
const turnBudget = (limits: { maxScreens: number; maxActions: number }): number => limits.maxActions * 3 + limits.maxScreens * 2 + 20;
/**
* A replay plan is checked here as well as by the CLI, for callers that pass one directly. It takes the model's place, so
* the options that only steer a model would do nothing; the plan's own fill and type steps name the secrets it types.
*/
function replayPlan(options: Pick<BinaryExploreOptions, "replay" | "provider" | "model" | "hints">): ReplayPlan | undefined {
if (!options.replay) return undefined;
const ignored = (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
if (ignored.length) {
throw binaryError("REPLAY_OPTIONS_IGNORED", `--replay takes the model's place, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
`Drop ${ignored.map(name => `--${name}`).join(" and ")}. A {{secret:NAME}} in the plan's fill and type steps is read from the environment, as one in --hints is.`);
}
const plan = parseReplayPlan(options.replay);
for (const [index, step] of plan.steps.entries()) {
if (!("action" in step) || step.action !== "open") continue;
try { assertAppUrl(step.url); }
catch (error) { throw new CiError({ code: "REPLAY_PLAN_INVALID", stage: "exploration", message: `Step ${index + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the replay plan." }); }
}
return plan;
}
/** A replay plan takes the model's place: no model access is checked or used. */
function replayModel(plan: ReplayPlan, platform: Platform, limits: { maxScreens: number; maxActions: number }) {
assertReplayFits(plan, platform, { ...limits, maxTurns: turnBudget(limits) });
const session = createReplaySession(plan, platform);
return { session, budgetNote: session.budgetNote, provider: "replay" as const, model: "replay plan", auth: "none" as const };
}
/**
* A flows file is checked here as well as by the CLI. It decides every step, as a replay plan does, but a step it
* describes in words is carried out by a model, so --provider, --model and --hints apply only when it has one.
*/
function flowsPlan(options: Pick<BinaryExploreOptions, "flows" | "replay" | "provider" | "model" | "hints">): FlowPlan | undefined {
if (!options.flows) return undefined;
if (options.replay) throw binaryError("FLOWS_WITH_REPLAY", "--flows and --replay each decide every step, so only one of them can run.", "Drop --replay, or move its steps into a flow.");
const plan = parseFlowPlan(options.flows);
const ignored = flowsNeedModel(plan) ? [] : (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
if (ignored.length) {
throw binaryError("FLOWS_OPTIONS_IGNORED", `Every step of the flows file is exact, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
`Drop ${ignored.map(name => `--${name}`).join(" and ")}, or describe a step in words ({"do": "…"}) for a model to carry out.`);
}
for (const [index, flow] of plan.flows.entries()) for (const [stepIndex, step] of flow.steps.entries()) {
if (!("action" in step) || step.action !== "open") continue;
try { assertAppUrl(step.url); }
catch (error) { throw new CiError({ code: "FLOWS_PLAN_INVALID", stage: "exploration", message: `Flow ${index + 1} (${flow.name}), step ${stepIndex + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the flows file." }); }
}
return plan;
}
/** The flows run in place of a model's exploration; a model is checked, and used, only when a step is described in words. */
async function flowsModel(plan: FlowPlan, platform: Platform, limits: { maxScreens: number; maxActions: number }, options: ModelSessionOptions & Pick<BinaryExploreOptions, "hints">,
environment: NodeJS.ProcessEnv, say: (message: string) => void, app: { name: string }) {
assertFlowsFit(plan, platform, { ...limits, maxTurns: turnBudget(limits) });
const goals = flowsNeedModel(plan, platform) ? await preflightModelSession(options, environment, say) : undefined;
const session = createFlowSession(plan, platform, { app, ...(goals ? { model: goals.session } : {}), ...(options.hints ? { hints: options.hints } : {}) });
return { session, budgetNote: session.budgetNote, flows: session.notReached, provider: "flows" as const, model: goals ? `${goals.provider} ${goals.model}` : session.model, auth: goals?.auth ?? "none" as const };
}
/** Local `claude`/`codex` login unless an API key is present; preflight proves access before a simulator is paid for. */
export function selectModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
const provider: CiAgentProviderName = options.provider ?? (environment.OPENAI_API_KEY?.trim() && !environment.ANTHROPIC_API_KEY?.trim() ? "openai" : "anthropic");
const keyName = provider === "anthropic" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY";
return modelSession(provider, environment[keyName]?.trim() ? "api-key" : "local-login", options, environment);
}
/**
* With --provider or --model, only that choice is tried. Otherwise every model access this shell has is tried
* in turn (API keys, then the local claude and codex logins) and the first that passes preflight runs.
*/
export async function preflightModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv, say: (message: string) => void) {
const candidates = options.provider || options.model ? [selectModelSession(options, environment)] : [
...(["anthropic", "openai"] as const).filter(provider => environment[provider === "anthropic" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY"]?.trim())
.map(provider => modelSession(provider, "api-key", options, environment)),
modelSession("anthropic", "local-login", options, environment),
modelSession("openai", "local-login", options, environment),
];
let firstError: unknown;
for (const [index, candidate] of candidates.entries()) {
say(`checking ${candidate.provider} ${candidate.model} (${candidate.auth === "api-key" ? "API key" : "local login"})`);
try {
await candidate.session.preflight();
return candidate;
} catch (error) {
firstError ??= error;
if (index < candidates.length - 1) say(`${candidate.provider} ${candidate.auth === "api-key" ? "API key" : "local login"} unavailable (${error instanceof Error ? error.message : String(error)}); trying the next`);
}
}
throw firstError;
}
export type SessionExploreOptions = Omit<BinaryExploreOptions, "archive" | "bundleId" | "owner" | "project"> & {
/** App to explore; defaults to the app in the foreground, else the only user-installed app. */
bundleId?: string;
@@ -329,8 +210,7 @@ export type ConnectedSessionExploration = {
limits: { maxDurationMinutes: number; maxScreens: number; maxActions: number };
store: ExploreStore;
map: ExploreMap;
/** `flows`: the session takes the explorer's flow tools, and names the checkpoints it never attempted. */
model: { session: Pick<CiAgentSession, "run">; budgetNote?: () => string | undefined; flows?: () => Array<{ flow: string; checkpoints: string[] }> };
strategy: ExplorationStrategy;
/** `eas simulator:get --json` for the attached session; descriptive (OS version) only. */
sessionJson: () => string;
/** agent-device session name on the daemon; explore:session reuses the caller's session so the device is not "in use" by another. */
@@ -353,7 +233,7 @@ export type ConnectedSessionExploration = {
/** Drives an agent-device daemon we already hold a connection to: optional install, open, then the model-led exploration. */
export async function exploreConnectedSession(ctx: ConnectedSessionExploration): Promise<{ runtime: ActiveSimulatorRuntime; exploration: BlackBoxExplorerResult }> {
const { connection, platform, device, bundleId, app, install, work, runner, environment, secrets, inputs, say, now, limits, store, map, model, sessionJson, launchUrl, hints, allowDestructive, local } = ctx;
const { connection, platform, device, bundleId, app, install, work, runner, environment, secrets, inputs, say, now, limits, store, map, strategy, sessionJson, launchUrl, hints, allowDestructive, local } = ctx;
const controllerSession = ctx.controllerSession ?? CONTROLLER_SESSION;
const controllerKind = connectionController(connection);
const controllerName = controllerKind === "argent" ? "Argent" : "agent-device";
@@ -392,7 +272,7 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
ctx.afterInstall?.();
// On a local device, the first open also builds agent-device's iOS runner.
// Flows start from the app's first screen: an app already running on an attached session is relaunched, not brought forward.
const opened = controller(["open", bundleId, ...(model.flows ? ["--relaunch"] : []), "--level", "full", "--json"], local ? 15 * 60_000 : 2 * 60_000);
const opened = controller(["open", bundleId, ...(strategy.flows ? ["--relaunch"] : []), "--level", "full", "--json"], local ? 15 * 60_000 : 2 * 60_000);
const openedDevice = openedSimulatorDevice(opened.stdout, bundleId, "initial app open", platform, controllerSession);
const data = controllerData(opened.stdout, "initial app open");
const deviceData = data.device && typeof data.device === "object" ? data.device as Record<string, unknown> : {};
@@ -539,12 +419,12 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
map.gaps.push({ id: `gap_${randomUUID().slice(0, 8)}`, kind: "coverage", status: "open", message: reason, createdAt: new Date(now()).toISOString() });
say("exploration blocked: the app does not stay running");
blockedAtLaunch = { stopReason: "blocked", summary: reason, screens: 0, observations: 0, actions: 0, turns: 0,
...(model.flows ? { flows: model.flows().map(({ flow, checkpoints }) => ({ name: flow, status: "failed" as const, reason, checkpoints: checkpoints.map(name => ({ name, missing: "The app did not stay open." })) })) } : {}) };
...(strategy.flows ? { flows: strategy.flows().map(({ flow, checkpoints }) => ({ name: flow, status: "failed" as const, reason, checkpoints: checkpoints.map(name => ({ name, missing: "The app did not stay open." })) })) } : {}) };
}
}
const exploration = blockedAtLaunch ?? await runBlackBoxExploration({ session: model.session, store, map, app: { name: app.name, bundleId, platform, ...(app.version ? { version: app.version } : {}) },
const exploration = blockedAtLaunch ?? await runBlackBoxExploration({ session: strategy.session, store, map, app: { name: app.name, bundleId, platform, ...(app.version ? { version: app.version } : {}) },
inputs, ...(hints ? { hints: hints } : {}), ...(launchUrl ? { launchUrl: launchUrl } : {}), ...(allowDestructive ? { allowDestructive: allowDestructive } : {}), ...(ctx.appIcon ? { appIcon: ctx.appIcon } : {}),
...(model.budgetNote ? { budgetNote: model.budgetNote } : {}), ...(model.flows ? { flows: { restart, notReached: model.flows } } : {}),
...(strategy.budgetNote ? { budgetNote: strategy.budgetNote } : {}), ...(strategy.flows ? { flows: { restart, notReached: strategy.flows } } : {}),
deadlineMs, maxScreens: limits.maxScreens, maxActions: limits.maxActions, maxTurns: turnBudget(limits), now, observe, progress: say,
act: async action => {
const perform = async () => {
@@ -590,6 +470,12 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
throw new Error(failureText(error));
}
} });
for (const planned of strategy.flowSteps?.() ?? []) {
const record = exploration.flows?.find(flow => flow.name.toLowerCase() === planned.flow.toLowerCase());
if (!record) continue;
record.steps = planned.steps;
if (planned.failedStep !== undefined && record.status === "failed") record.failedStep = planned.failedStep;
}
// "complete" is reserved for a run with no pending frontier or open gaps; a model that finishes early is "partial".
// The map schema allows stopReason only on partial/failed runs.
map.run.status = exploration.stopReason === "model-error" ? "failed"
@@ -609,11 +495,10 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
const environment = options.environment ?? process.env;
const runner = options.runner ?? defaultRunner;
const limits = validatedLimits(options);
const replay = replayPlan(options);
const flows = flowsPlan(options);
const choice = chooseStrategy(options);
// A flows run without --platform checks its secrets once the session reports its platform, before any device work, so a
// secret that only another platform's steps type is not required.
let inputs = flows && !options.platform ? undefined : hintInputs(options.hints, environment, replay, flows, options.platform);
let inputs = choice.kind === "flows" && !options.platform ? undefined : strategyInputs(choice, options.hints, environment, options.platform);
if (options.bundleId !== undefined && !appIdPattern.test(options.bundleId)) throw binaryError("BUNDLE_ID_INVALID", `${options.bundleId} is not a bundle or package id.`, "Pass --bundle-id with the app's bundle identifier (iOS) or package name (Android).");
let connection: SimulatorConnection;
try { connection = await resolveAttachedSimulatorSession(environment, options.envFile, options.session); }
@@ -659,7 +544,7 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
}
platform = info.platform;
if (!inputs) {
inputs = hintInputs(options.hints, environment, replay, flows, platform);
inputs = strategyInputs(choice, options.hints, environment, platform);
secrets.push(...inputs.values);
}
const candidates = daemonSessions(runner, connection, work, platform, environment).filter(item => !device || item.device === device);
@@ -681,16 +566,15 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
map.run.environment = { deviceModel: device };
session = { id: connection.sessionId, platform, device, name: info.name, ...(info.deviceRunSessionUrl ? { url: info.deviceRunSessionUrl } : {}) };
const model = replay ? replayModel(replay, platform, limits) : flows ? await flowsModel(flows, platform, limits, options, environment, say, app)
: await preflightModelSession(options, environment, say);
const strategy = await startStrategy(choice, { platform, limits, options, environment, say, app });
const outputDirectory = await realpath(await mkdir(options.output, { recursive: true }).then(() => options.output));
const store = new ExploreStore(outputDirectory);
if (await store.exists()) throw binaryError("OUTPUT_NOT_EMPTY", `${outputDirectory} already holds an Explore map.`, "Pass an empty --output directory.");
({ runtime, exploration } = await exploreConnectedSession({ connection, platform, device, bundleId: app.bundleId, app, work, runner, environment, secrets, inputs, say, now, limits, store, map, model, sessionJson,
({ runtime, exploration } = await exploreConnectedSession({ connection, platform, device, bundleId: app.bundleId, app, work, runner, environment, secrets, inputs, say, now, limits, store, map, strategy, sessionJson,
...(existing ? { controllerSession: existing.name } : {}),
...(options.launchUrl ? { launchUrl: options.launchUrl } : {}), ...(options.hints ? { hints: options.hints } : {}), ...(options.allowDestructive ? { allowDestructive: options.allowDestructive } : {}) }));
return { outputDirectory, mapPath: store.mapPath, app: { ...app, platform }, session, runtime, model: { provider: model.provider, model: model.model, auth: model.auth }, exploration, durationMs: Date.now() - startedAt };
return { outputDirectory, mapPath: store.mapPath, app: { ...app, platform }, session, runtime, model: { provider: strategy.provider, model: strategy.model, auth: strategy.auth }, exploration, durationMs: Date.now() - startedAt };
} catch (error) {
primaryError = toCiError(error, { code: "EXPLORE_SESSION_FAILED", stage: "exploration", nextAction: "Inspect the diagnostic, correct the failing prerequisite, and rerun explore:session. The session was left running." });
throw primaryError;
@@ -775,11 +659,10 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
const environment = options.environment ?? process.env;
const runner = options.runner ?? defaultRunner;
const limits = validatedLimits(options);
const replay = replayPlan(options);
const flows = flowsPlan(options);
const choice = chooseStrategy(options);
// The archive's name gives its platform before it is read (inspectArchive confirms it), so a secret only another platform's steps type is not required.
const namedPlatform = options.platform ?? (() => { try { return archiveKind(options.archive) === "android-apk" ? "android" as const : "ios" as const; } catch { return undefined; } })();
const inputs = hintInputs(options.hints, environment, replay, flows, namedPlatform);
const inputs = strategyInputs(choice, options.hints, environment, namedPlatform);
const local = options.simulator === "local";
// A local emulator runs this Mac's ABI (arm64-v8a on Apple silicon); EAS's runs x86_64.
const emulatorAbi = local ? hostEmulatorAbi() : "x86_64";
@@ -796,8 +679,7 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
say(`${archive.appName} ${archive.version ?? ""} (${bundleId}, ${platform}, ${(archive.bytes / 1048576).toFixed(0)} MB, sha256 ${archive.sha256.slice(0, 12)})`);
if (archive.missingLibs?.length && emulatorAbi === "x86_64") say(`warning: lib/x86_64 lacks ${archive.missingLibs.slice(0, 4).join(", ")}${archive.missingLibs.length > 4 ? ` and ${archive.missingLibs.length - 4} more` : ""} that another ABI ships; the app may stop with UnsatisfiedLinkError on the emulator`);
const model = replay ? replayModel(replay, platform, limits) : flows ? await flowsModel(flows, platform, limits, options, environment, say, { name: archive.appName })
: await preflightModelSession(options, environment, say);
const strategy = await startStrategy(choice, { platform, limits, options, environment, say, app: { name: archive.appName } });
const outputDirectory = await realpath(await mkdir(options.output, { recursive: true }).then(() => options.output));
const store = new ExploreStore(outputDirectory);
@@ -837,7 +719,7 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
...(appIcon ? { appIcon } : {}), app: { name: archive.appName, ...(archive.version ? { version: archive.version } : {}) },
// A local device has the app installed by simctl or adb before the controller starts.
...(localDevice ? { local: localDevice } : { install: { path: installPath, kind: archive.kind } }),
work, runner, environment, secrets: [environment.EXPO_TOKEN ?? "", connection.daemonAuthToken, ...inputs.values].filter(Boolean), inputs, say, now, limits, store, map, model,
work, runner, environment, secrets: [environment.EXPO_TOKEN ?? "", connection.daemonAuthToken, ...inputs.values].filter(Boolean), inputs, say, now, limits, store, map, strategy,
urlSchemes: archive.urlSchemes, sessionJson,
...(options.launchUrl ? { launchUrl: options.launchUrl } : {}), ...(options.hints ? { hints: options.hints } : {}), ...(options.allowDestructive ? { allowDestructive: options.allowDestructive } : {}) });
if (local) {
@@ -920,7 +802,7 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
}
if (primaryError) throw primaryError;
return { outputDirectory, mapPath: store.mapPath, archive, session: { id: sessionId!, platform, device, owner, project }, runtime: runtime!,
model: { provider: model.provider, model: model.model, auth: model.auth }, exploration: exploration!, durationMs: Date.now() - startedAt };
model: { provider: strategy.provider, model: strategy.model, auth: strategy.auth }, exploration: exploration!, durationMs: Date.now() - startedAt };
}
function lastLine(text: string): string { return text.trim().split(/\r?\n/).filter(Boolean).at(-1) ?? ""; }
@@ -935,10 +817,6 @@ async function easLoggedIn(runner: typeof defaultRunner, environment: NodeJS.Pro
} catch { return false; }
}
/** Schemes the OS routes to a browser, the store, Settings, or another app: opening one hands the session away from the app under test. */
const SYSTEM_URL_SCHEMES = /^(https?|ftp|file|javascript|data|blob|mailto|tel|telprompt|facetime(-audio)?|sms|mms|imessage|itms(-apps|-services|-beta)?|macappstore|app-settings|prefs|shortcuts|maps|music|videos|photos-redirect|x-apple-[a-z-]+|market|intent|content|android-app|geo|package):$/i;
/** The model may only open app-owned URLs; an https page or a store link would hand the session to another app. */
const RUNNER_BUSY = /still finishing a previous command/i;
const sleepSync = (ms: number): void => { Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); };
@@ -957,16 +835,6 @@ export function retryWhileBusy<T>(run: () => T, pause: (ms: number) => void, bud
}
}
export function assertAppUrl(url: string, appSchemes: readonly string[] = []): void {
let parsed: URL;
try { parsed = new URL(url); } catch { throw binaryError("AGENT_URL_INVALID", `Not a URL: ${url}`, "Pass a custom-scheme app URL."); }
if (SYSTEM_URL_SCHEMES.test(parsed.protocol)) throw binaryError("AGENT_URL_NOT_ALLOWED", `${parsed.protocol} URLs leave the app; only the app's own URL schemes are allowed.`, "Use the app's own URL scheme.");
const scheme = parsed.protocol.slice(0, -1).toLowerCase();
if (appSchemes.length && !appSchemes.includes(scheme)) {
throw binaryError("AGENT_URL_NOT_ALLOWED", `${parsed.protocol} is not a URL scheme this app registers (${appSchemes.join(", ")}).`, "Use one of the app's own URL schemes.");
}
}
export const BLANK_SCREENSHOT_MESSAGE = "The app blocks screen capture on this screen (Android FLAG_SECURE), so its screenshot came back blank.";
export { solidColour };
+175
View File
@@ -0,0 +1,175 @@
import { createCiAgentSession, type CiAgentSession } from "../capture/agent/session.js";
import { createLocalCiAgentSession } from "../capture/agent/session-local.js";
import type { CiAgentProviderName } from "../capture/agent/provider.js";
import { createExplorationInputs } from "../config/exploration-inputs.js";
import type { ExplorationFlowStep } from "../contracts/standalone-report.js";
import { CiError } from "../shared/ci-errors.js";
import type { Platform } from "../types.js";
import { assertAppUrl } from "./app-url.js";
import { assertFlowsFit, createFlowSession, flowGoals, flowSetup, flowsNeedModel, parseFlowPlan, type FlowPlan } from "./flows.js";
import { assertReplayFits, createReplaySession, parseReplayPlan, replaySetup, type ReplayPlan } from "./replay.js";
import type { BinaryExploreOptions } from "./run.js";
/**
* How an exploration decides what to do on the device. The explorer (`runBlackBoxExploration`) owns the device and the
* tools that drive it, the budgets, what is refused, and what is recorded; a strategy only calls those tools, through
* `session.run`. A model explores on its own, a replay plan takes fixed steps, and a flows file takes named flows.
* A new strategy is one more case here: what `chooseStrategy` checks before any device starts, and the session
* `startStrategy` runs.
*/
export type StrategyChoice = { kind: "model" } | { kind: "replay"; plan: ReplayPlan } | { kind: "flows"; plan: FlowPlan };
export type ModelAuth = "local-login" | "api-key";
/** A strategy ready to run, and how the report names it. */
export type ExplorationStrategy = {
session: Pick<CiAgentSession, "run">;
provider: CiAgentProviderName | "replay" | "flows";
model: string;
auth: ModelAuth | "none";
/** A sentence a budget stop's gap adds to its own: the planned screens or checkpoints the run did not reach. */
budgetNote?: () => string | undefined;
/**
* Takes the explorer's flow tools (start_flow, checkpoint, end_flow), with a relaunch before each flow after the first,
* and names each flow's checkpoints it never attempted, for the explorer to record as missing.
*/
flows?: () => Array<{ flow: string; checkpoints: string[] }>;
/** Each flow's steps and the one it failed at, which the report keeps beside the flow. */
flowSteps?: () => Array<{ flow: string; steps: ExplorationFlowStep[]; failedStep?: number }>;
};
export const DEFAULT_MODELS: Record<CiAgentProviderName, string> = { anthropic: "claude-sonnet-4-5", openai: "gpt-6.1-sol" };
type StrategyOptions = Pick<BinaryExploreOptions, "replay" | "flows" | "provider" | "model" | "hints" | "createSession">;
type ModelSessionOptions = Pick<BinaryExploreOptions, "provider" | "model" | "createSession">;
function strategyError(code: string, message: string, nextAction: string): CiError {
return new CiError({ code, stage: "session", message, nextAction });
}
/** The turns the model gets: three for each action and two for each screen, plus a few to look around and finish. */
export const turnBudget = (limits: { maxScreens: number; maxActions: number }): number => limits.maxActions * 3 + limits.maxScreens * 2 + 20;
/**
* The strategy the options ask for, checked before any device starts. A plan file is checked here as well as by the
* CLI, for callers that pass one directly.
*/
export function chooseStrategy(options: StrategyOptions): StrategyChoice {
if (options.replay && options.flows) throw strategyError("FLOWS_WITH_REPLAY", "--flows and --replay each decide every step, so only one of them can run.", "Drop --replay, or move its steps into a flow.");
if (options.replay) return { kind: "replay", plan: replayPlan(options.replay, options) };
if (options.flows) return { kind: "flows", plan: flowsPlan(options.flows, options) };
return { kind: "model" };
}
/**
* A replay plan takes the model's place, so the options that only steer a model would do nothing; the plan's own fill
* and type steps name the secrets it types.
*/
function replayPlan(value: ReplayPlan, options: StrategyOptions): ReplayPlan {
const ignored = (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
if (ignored.length) {
throw strategyError("REPLAY_OPTIONS_IGNORED", `--replay takes the model's place, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
`Drop ${ignored.map(name => `--${name}`).join(" and ")}. A {{secret:NAME}} in the plan's fill and type steps is read from the environment, as one in --hints is.`);
}
const plan = parseReplayPlan(value);
for (const [index, step] of plan.steps.entries()) {
if (!("action" in step) || step.action !== "open") continue;
try { assertAppUrl(step.url); }
catch (error) { throw new CiError({ code: "REPLAY_PLAN_INVALID", stage: "exploration", message: `Step ${index + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the replay plan." }); }
}
return plan;
}
/**
* A flows file decides every step, as a replay plan does, but a step it describes in words is carried out by a model,
* so --provider, --model and --hints apply only when it has one.
*/
function flowsPlan(value: FlowPlan, options: StrategyOptions): FlowPlan {
const plan = parseFlowPlan(value);
const ignored = flowsNeedModel(plan) ? [] : (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
if (ignored.length) {
throw strategyError("FLOWS_OPTIONS_IGNORED", `Every step of the flows file is exact, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
`Drop ${ignored.map(name => `--${name}`).join(" and ")}, or describe a step in words ({"do": "…"}) for a model to carry out.`);
}
for (const [index, flow] of plan.flows.entries()) for (const [stepIndex, step] of flow.steps.entries()) {
if (!("action" in step) || step.action !== "open") continue;
try { assertAppUrl(step.url); }
catch (error) { throw new CiError({ code: "FLOWS_PLAN_INVALID", stage: "exploration", message: `Flow ${index + 1} (${flow.name}), step ${stepIndex + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the flows file." }); }
}
return plan;
}
/**
* App-test inputs: each {{secret:NAME}} in the hints, and in the strategy's own fill and type steps and described steps,
* names an environment variable whose value Explore types. Read before any session starts, so a missing one costs
* nothing; a step limited to another platform than `platform`, when it is known, needs none.
*/
export function strategyInputs(choice: StrategyChoice, hints: string | undefined, environment: NodeJS.ProcessEnv, platform?: Platform): ReturnType<typeof createExplorationInputs> {
const setup = choice.kind === "replay" ? replaySetup(choice.plan) : choice.kind === "flows" ? flowSetup(choice.plan, platform) : undefined;
try { return createExplorationInputs({ instructions: [hints ?? "", ...(choice.kind === "flows" ? flowGoals(choice.plan, platform) : [])].join("\n"), ...(setup ? { setup } : {}), environment }); }
catch (error) {
if (!(error instanceof CiError) || !error.code.startsWith("EXPLORATION_SECRET_")) throw error;
const name = typeof error.details?.secretName === "string" ? error.details.secretName : "NAME";
throw new CiError({ code: error.code, stage: "preflight", message: error.message, ...(error.details ? { details: error.details } : {}),
nextAction: error.code === "EXPLORATION_SECRET_REFERENCE_INVALID"
? "Write each input in --hints as {{secret:NAME}} with an uppercase environment variable name that is not a provider, Expo, or CI credential."
: `Set ${name} in the environment that runs Explore (export ${name}=…) to the single-line test value, then rerun.` });
}
}
/** The session the chosen strategy runs. Model access is checked here, before a simulator is paid for, and only when the strategy uses a model. */
export async function startStrategy(choice: StrategyChoice, context: { platform: Platform; limits: { maxScreens: number; maxActions: number }; options: StrategyOptions;
environment: NodeJS.ProcessEnv; say: (message: string) => void; app: { name: string } }): Promise<ExplorationStrategy> {
const { platform, limits, options, environment, say, app } = context;
if (choice.kind === "replay") {
assertReplayFits(choice.plan, platform, { ...limits, maxTurns: turnBudget(limits) });
const session = createReplaySession(choice.plan, platform);
return { session, budgetNote: session.budgetNote, provider: "replay", model: "replay plan", auth: "none" };
}
if (choice.kind === "flows") {
assertFlowsFit(choice.plan, platform, { ...limits, maxTurns: turnBudget(limits) });
const goals = flowsNeedModel(choice.plan, platform) ? await preflightModelSession(options, environment, say) : undefined;
const session = createFlowSession(choice.plan, platform, { app, ...(goals ? { model: goals.session } : {}), ...(options.hints ? { hints: options.hints } : {}) });
return { session, budgetNote: session.budgetNote, flows: session.notReached, flowSteps: session.steps, provider: "flows", model: goals ? `${goals.provider} ${goals.model}` : session.model, auth: goals?.auth ?? "none" };
}
return preflightModelSession(options, environment, say);
}
function modelSession(provider: CiAgentProviderName, auth: ModelAuth, options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
const config = { provider, model: options.model ?? DEFAULT_MODELS[provider] };
const session = options.createSession ? options.createSession(config, { env: environment, auth })
: auth === "api-key" ? createCiAgentSession(config, { env: environment }) : createLocalCiAgentSession(config, { env: environment });
return { session, provider, model: config.model, auth };
}
/** Local `claude`/`codex` login unless an API key is present; preflight proves access before a simulator is paid for. */
export function selectModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
const provider: CiAgentProviderName = options.provider ?? (environment.OPENAI_API_KEY?.trim() && !environment.ANTHROPIC_API_KEY?.trim() ? "openai" : "anthropic");
const keyName = provider === "anthropic" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY";
return modelSession(provider, environment[keyName]?.trim() ? "api-key" : "local-login", options, environment);
}
/**
* With --provider or --model, only that choice is tried. Otherwise every model access this shell has is tried
* in turn (API keys, then the local claude and codex logins) and the first that passes preflight runs.
*/
export async function preflightModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv, say: (message: string) => void) {
const candidates = options.provider || options.model ? [selectModelSession(options, environment)] : [
...(["anthropic", "openai"] as const).filter(provider => environment[provider === "anthropic" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY"]?.trim())
.map(provider => modelSession(provider, "api-key", options, environment)),
modelSession("anthropic", "local-login", options, environment),
modelSession("openai", "local-login", options, environment),
];
let firstError: unknown;
for (const [index, candidate] of candidates.entries()) {
say(`checking ${candidate.provider} ${candidate.model} (${candidate.auth === "api-key" ? "API key" : "local login"})`);
try {
await candidate.session.preflight();
return candidate;
} catch (error) {
firstError ??= error;
if (index < candidates.length - 1) say(`${candidate.provider} ${candidate.auth === "api-key" ? "API key" : "local login"} unavailable (${error instanceof Error ? error.message : String(error)}); trying the next`);
}
}
throw firstError;
}
+1 -1
View File
@@ -3,7 +3,7 @@ import { existsSync } from "node:fs";
import { mkdir, readdir, readFile, rm, stat, symlink, writeFile } from "node:fs/promises";
import { homedir } from "node:os";
import { join } from "node:path";
import { apkAbis } from "../binary/archive.js";
import { apkAbis } from "../shared/archive-entries.js";
import { exploreCacheRoot } from "../shared/cache-root.js";
import { CiError } from "../shared/ci-errors.js";
import { isRecord, parseJson } from "./common.js";
+1 -1
View File
@@ -7,7 +7,7 @@ import { CiError, environmentSecrets } from "../shared/ci-errors.js";
import { silentProgress, type Progress } from "../shared/progress.js";
import { githubRunLink, safeText, silentTimings, type Timings } from "../shared/timings.js";
import { retryGitHubLookup } from "../report/github-lookup.js";
import { safeRelativePath } from "../binary/archive.js";
import { safeRelativePath } from "../shared/archive-entries.js";
import { CAPTURE_CACHE_MAX_BYTES, CAPTURE_CACHE_MAX_FILES } from "./artifact-cache.js";
import { baselineLookup, githubContext, trustedBaselineRun, type GithubArtifact, type GithubRun } from "./workflow-steps.js";
+2 -1
View File
@@ -11,7 +11,7 @@ import { help } from "./cli/help.js";
import { listScreens, getScreen, listElements, listObservations, getObservation, nextFrontier, listAttempts, getAttempt, listTransitions, getTransition, neighbors, pathQuery, listJourneys, getJourney, listFrontier, listGaps, summary, search, status, validate, exportArtifacts } from "./cli/map-commands.js";
import { compareCommand, exportWeb, view } from "./cli/catalog-commands.js";
import { ciReport, ciPublish, ciFailure, ciDeploy, ciPullRequest, ciFindBaseline, ciCompareRefs, ciAfterCapture, setup, doctor, sourceVerify, captureReview, captureBaseline, configPlatforms, configTargetList } from "./cli/ci-commands.js";
import { exploreBinary, exploreSession } from "./cli/binary-commands.js";
import { compareFlows, exploreBinary, exploreSession } from "./cli/binary-commands.js";
import { reports, reportUpload } from "./cli/report-commands.js";
const handlers: Record<string, () => Promise<void> | void> = {
@@ -57,6 +57,7 @@ const handlers: Record<string, () => Promise<void> | void> = {
"config:targets": configTargetList,
doctor,
compare: compareCommand,
"compare:flows": compareFlows,
view,
help,
"--help": help,
+2
View File
@@ -19,6 +19,7 @@ const aliases: Record<string, string> = {
"frontier list": "frontier:list",
"gap list": "gap:list",
"export web": "export:web",
"compare flows": "compare:flows",
"ci report": "ci:report",
"ci publish": "ci:publish",
"ci failure": "ci:failure",
@@ -104,6 +105,7 @@ const flagKinds: Record<string, Record<string, "value" | "boolean">> = {
"ci:find-baseline": { sha:"value", target:"value", platform:"value" },
"ci:compare-refs": { project:"value", base:"value", head:"value", title:"value", output:"value" },
"ci:after-capture": { output:"value", checkout:"value" },
"compare:flows": { before: "value", after: "value", output: "value", report: "value" },
compare: {
baseline: "value",
current: "value",
+41 -3
View File
@@ -1,4 +1,4 @@
import { stat } from "node:fs/promises";
import { readFile, stat, writeFile } from "node:fs/promises";
import { basename, join, resolve } from "node:path";
import { DEFAULT_BINARY_PROJECT, runBinaryExploration, runSessionExploration } from "../binary/run.js";
import { openViewerInBrowser, startViewerServer, waitForViewerShutdown } from "../map/viewer-server.js";
@@ -6,10 +6,15 @@ import { writeExplorationReport } from "../binary/report.js";
import { assertHostingExport, deployExplorationReport, zipReportDirectory } from "../report/preview-deploy.js";
import { keepReports } from "./report-commands.js";
import { warnIgnoredExpiry } from "../config/explore-config.js";
import { argumentError, flag, flags, has, nonNegativeInteger, oneOf, output, positional, positiveInteger } from "./args.js";
import { argumentError, flag, flags, has, nonNegativeInteger, oneOf, output, positional, positiveInteger, required } from "./args.js";
import { readReplayPlan } from "../binary/replay.js";
import { readFlowPlan } from "../binary/flows.js";
import type { ExplorationFlow } from "../contracts/standalone-report.js";
import { EXPLORATION_REPORT_KIND, type ExplorationFlow, type ExplorationReport } from "../contracts/standalone-report.js";
import { compareFlowRuns, type FlowRun } from "../review/flow-comparison.js";
import { writeFlowReviewReport } from "../review/flow-review.js";
import { diffScreenshots } from "../review/visual-evidence.js";
import { agentDeviceLauncher, ensureController } from "../shared/controller-cache.js";
import { CiError } from "../shared/ci-errors.js";
/** `--alias NAME` plus a fixed token from EXPLORE_PREVIEW_TOKEN: the hosted link stays the same from one run to the next (validated by the deploy). */
@@ -156,3 +161,36 @@ export async function exploreSession(): Promise<void> {
openViewerInBrowser(viewer.url);
await waitForViewerShutdown(viewer);
}
/** A flows run's report: the report directory explore:binary or explore:session wrote, or the output directory holding it. */
async function flowRun(path: string, name: "before" | "after"): Promise<FlowRun> {
const refuse = (message: string) => { throw new CiError({ code: "FLOW_COMPARE_INPUT_INVALID", stage: "preflight", message,
nextAction: `Pass --${name} the output directory of an explore:binary or explore:session run that used --flows (or its report directory).` }); };
for (const directory of [resolve(path), resolve(path, "report")]) {
let text: string;
try { text = await readFile(join(directory, "exploration.json"), "utf8"); } catch { continue; }
let report: ExplorationReport;
try { report = JSON.parse(text) as ExplorationReport; } catch { return refuse(`${join(directory, "exploration.json")} is not JSON.`); }
if (report?.kind !== EXPLORATION_REPORT_KIND) return refuse(`${join(directory, "exploration.json")} is not an exploration report.`);
if (!Array.isArray(report.exploration?.flows)) return refuse(`The run in ${directory} did not follow a flows file, so it has no checkpoints to compare.`);
return { report, directory };
}
return refuse(`No exploration.json was found in ${resolve(path)} or its report folder.`);
}
/**
* `compare:flows --before DIR --after DIR [--output FILE] [--report DIR]`: two runs of one flows file, checkpoint by
* checkpoint. `--report` writes a report the Viewer opens on its Flows view.
*/
export async function compareFlows(): Promise<void> {
const [before, after] = await Promise.all([flowRun(required("before"), "before"), flowRun(required("after"), "after")]);
let command: string | undefined;
const comparison = await compareFlowRuns(before, after, async (beforePath, afterPath) => {
if (!command) { await ensureController("agent-device", { environment: process.env }); command = agentDeviceLauncher(process.env); }
return diffScreenshots({ baselinePath: beforePath, currentPath: afterPath, command });
});
const path = flag("output"), reportPath = flag("report");
if (path) await writeFile(resolve(path), JSON.stringify(comparison, null, 2));
const report = reportPath ? await writeFlowReviewReport(comparison, before, after, reportPath) : undefined;
output({ ok: true, ...(path ? { output: resolve(path) } : {}), ...(report ? { report } : {}), comparison });
}
+3
View File
@@ -84,6 +84,9 @@ const CI = `CI (the installed workflows run these; they manage remote EAS Simula
capture:baseline --project DIRECTORY --sha COMMIT --output TEMP_DIRECTORY [--config FILE] [--target ID | --platform ios|android] [--controller agent-device|argent] [--simulator eas|local] [--repack eas|local]
source:verify --base-project DIRECTORY --head-project DIRECTORY
--base-sha COMMIT --head-sha COMMIT --output DIRECTORY [--config FILE] [--scope all|changes] [--target ID | --platform ios|android]
compare:flows --before DIRECTORY --after DIRECTORY [--output FILE] [--report DIRECTORY]
(two runs of one --flows file, checkpoint by checkpoint: unchanged, changed with where, or why not compared;
--report writes a report the Viewer opens on its Flows view)
compare --baseline DIRECTORY --current DIRECTORY --output FILE
[--review-output FILE] [--catalog-output FILE] [--catalog-name NAME]
[--base-surface-coverage FILE] [--head-surface-coverage FILE]
+1 -1
View File
@@ -189,7 +189,7 @@ export const DUO_PANELS = {
inner: { width: 951, height: 669, statusBar: { x: 883, y: 34, width: 40, height: 64 } },
} as const;
const DUO_STATUS_BAR_PADDING_POINTS = 4;
const isFoldable = (platform: "ios" | "android" | undefined, device: string | undefined) => platform !== "android" && /\bduo\b/i.test(device ?? "");
export const isFoldable = (platform: "ios" | "android" | undefined, device: string | undefined) => platform !== "android" && /\bduo\b/i.test(device ?? "");
/**
* The areas the verdict leaves out for a capture: the platform's status-bar band and, on iOS, its home-indicator
+86
View File
@@ -0,0 +1,86 @@
import type { ExplorationFlow, ExplorationFlowStep, ExplorationReport } from "./standalone-report.js";
export const FLOW_COMPARISON_KIND = "explore.flow-comparison" as const;
/**
* `changed` and `unchanged` compare the two screenshots; `different-size` could not (another device or orientation);
* `missing` has no screenshot on one side; `added` and `removed` are in only one run's flows file.
*/
export type CheckpointVerdict = "unchanged" | "changed" | "different-size" | "missing" | "added" | "removed";
/** Where two screenshots differ, in percent of the image (0–100). */
export type ChangedRegion = { x: number; y: number; width: number; height: number };
/** One build's side of a checkpoint in a flow review: its screenshot (an asset of the review's report) and the text its screen showed, or why it has none. */
export type FlowReviewSide = { screenshot: string; text?: string[] } | { missing: string };
export type FlowReviewCheckpoint = { name: string; verdict: CheckpointVerdict; before?: FlowReviewSide; after?: FlowReviewSide; regions?: ChangedRegion[] };
/** How one build's run took the flow. */
export type FlowReviewRun = Pick<ExplorationFlow, "status" | "reason" | "failedStep"> & { steps?: ExplorationFlowStep[] };
export type FlowReviewFlow = { name: string; before?: FlowReviewRun; after?: FlowReviewRun; checkpoints: FlowReviewCheckpoint[] };
/**
* Two runs of one flows file, compared checkpoint by checkpoint, as the Viewer shows them: carried in the after run's
* exploration report, whose assets hold both builds' checkpoint screenshots.
*/
export type FlowReview = {
before: RunConditions;
after: RunConditions;
/** How the runs were captured differently; a checkpoint that changed may owe its change to these. */
conditions: RunConditionDifference[];
notes: string[];
flows: FlowReviewFlow[];
};
/** One run's side of a checkpoint: its screenshot (a path inside that run's report directory), or why it has none. */
export type CheckpointSide = { observationId: string; screenshot: string } | { missing: string };
/**
* A checkpoint of one flow in two runs, paired by flow name and checkpoint name. `before` or `after` is absent when
* that run's flows file has no such flow or checkpoint.
*/
export type CheckpointPair = { flow: string; checkpoint: string; before?: CheckpointSide; after?: CheckpointSide };
/** What a run was captured on: two runs that differ here can still be compared, but a difference may come from it. */
export type RunConditions = { app: string; platform: string; device: string; osVersion?: string; locale?: string; orientation?: string; colorScheme?: string };
export type RunConditionDifference = { field: keyof RunConditions; before: string; after: string };
export function runConditions(report: ExplorationReport): RunConditions {
const environment = report.map.run.environment ?? {};
return { app: report.exploration.app.bundleId, platform: report.exploration.app.platform, device: report.exploration.device.model,
...(report.exploration.device.osVersion ? { osVersion: report.exploration.device.osVersion } : {}),
...(environment.locale ? { locale: environment.locale } : {}), ...(environment.orientation ? { orientation: environment.orientation } : {}),
...(environment.colorScheme ? { colorScheme: environment.colorScheme } : {}) };
}
export function conditionDifferences(before: RunConditions, after: RunConditions): RunConditionDifference[] {
const fields = ["app", "platform", "device", "osVersion", "locale", "orientation", "colorScheme"] as const;
return fields.flatMap(field => (before[field] ?? "unknown") === (after[field] ?? "unknown") ? []
: [{ field, before: before[field] ?? "unknown", after: after[field] ?? "unknown" }]);
}
const keyOf = (name: string) => name.trim().toLowerCase();
function side(report: ExplorationReport, checkpoint: NonNullable<ExplorationReport["exploration"]["flows"]>[number]["checkpoints"][number]): CheckpointSide {
if (checkpoint.observationId === undefined) return { missing: checkpoint.missing };
const observation = report.map.observations.find(item => item.id === checkpoint.observationId);
const artifact = observation?.screenshotArtifactId ? report.map.artifacts.find(item => item.id === observation.screenshotArtifactId) : undefined;
const asset = artifact ? report.assets.find(item => item.sha256 === artifact.sha256) : undefined;
return asset ? { observationId: checkpoint.observationId, screenshot: asset.path } : { missing: "The report has no screenshot for this checkpoint." };
}
/**
* Pairs the checkpoints of two runs of a flows file: the after run's flows and checkpoints in its order, then those only
* the before run has. Names match as the flows file compares them, ignoring case and surrounding space.
*/
export function pairCheckpoints(before: ExplorationReport, after: ExplorationReport): CheckpointPair[] {
const beforeFlows = before.exploration.flows ?? [], afterFlows = after.exploration.flows ?? [];
const flowNames = [...afterFlows, ...beforeFlows.filter(flow => !afterFlows.some(other => keyOf(other.name) === keyOf(flow.name)))].map(flow => flow.name);
return flowNames.flatMap(name => {
const was = beforeFlows.find(flow => keyOf(flow.name) === keyOf(name)), now = afterFlows.find(flow => keyOf(flow.name) === keyOf(name));
const wasPoints = was?.checkpoints ?? [], nowPoints = now?.checkpoints ?? [];
const names = [...nowPoints, ...wasPoints.filter(point => !nowPoints.some(other => keyOf(other.name) === keyOf(point.name)))].map(point => point.name);
return names.map(checkpoint => {
const old = wasPoints.find(point => keyOf(point.name) === keyOf(checkpoint)), current = nowPoints.find(point => keyOf(point.name) === keyOf(checkpoint));
return { flow: name, checkpoint, ...(old ? { before: side(before, old) } : {}), ...(current ? { after: side(after, current) } : {}) };
});
});
}
+10
View File
@@ -1,4 +1,5 @@
import type { PublicReview } from "./capture.js";
import type { FlowReview } from "./flow-comparison.js";
import type { BuildWorkspacePayload, Coverage, ExploreMap } from "../types.js";
export const STANDALONE_REPORT_CONTRACT = "expo.explore.standalone-report.v1" as const;
@@ -24,8 +25,15 @@ export type ExplorationFlow = {
/** The map's journeys for the flow: its walk from the screen it started on, in parts where the app moved without a transition. */
journeyIds?: string[];
checkpoints: Array<{ name: string; observationId: string; missing?: never } | { name: string; missing: string; observationId?: never }>;
/** The flow's steps on this run's platform, as the flows file names them. */
steps?: ExplorationFlowStep[];
/** The step the flow failed at, counted from 1 in `steps`. */
failedStep?: number;
};
/** An exact action (`tap "Settings"`), a step described in words for a model, or a checkpoint. */
export type ExplorationFlowStep = { action: string } | { do: string } | { checkpoint: string };
/** What `explore:binary` learned about one prebuilt app: provenance plus the map it explored, with no review or pull request. */
export type ExplorationReport = {
kind: typeof EXPLORATION_REPORT_KIND;
@@ -47,6 +55,8 @@ export type ExplorationReport = {
exploreVersion: string;
};
assets: Array<{ sha256: string; path: string }>;
/** Present when compare:flows wrote this report: the run compared with an earlier run of the same flows file. */
flowReview?: FlowReview;
};
export type EmbeddedReport = {
+77
View File
@@ -0,0 +1,77 @@
import { open } from "node:fs/promises";
import { resolve } from "node:path";
import { isFoldable, systemRegions } from "../contracts/capture.js";
import {
FLOW_COMPARISON_KIND, conditionDifferences, pairCheckpoints, runConditions,
type ChangedRegion, type CheckpointPair, type CheckpointVerdict, type RunConditionDifference, type RunConditions,
} from "../contracts/flow-comparison.js";
import type { ExplorationReport } from "../contracts/standalone-report.js";
import { ignoreSystemRegions, systemBandFrameLoader } from "./evidence-assessment.js";
import type { ScreenshotDiffMeasurement } from "./visual-evidence.js";
export type { CheckpointVerdict } from "../contracts/flow-comparison.js";
export type ComparedCheckpoint = CheckpointPair & {
verdict: CheckpointVerdict;
/** Share of the screenshot that differs once the status bar and home indicator are set aside. */
mismatchPercentage?: number;
regions?: ChangedRegion[];
};
export type FlowComparison = {
kind: typeof FLOW_COMPARISON_KIND;
before: RunConditions & { report: string };
after: RunConditions & { report: string };
/** How the runs were captured differently; a checkpoint that changed may owe its change to these. */
conditions: RunConditionDifference[];
notes: string[];
counts: Record<CheckpointVerdict, number>;
checkpoints: ComparedCheckpoint[];
};
export type FlowRun = { report: ExplorationReport; directory: string };
export type MeasureScreenshots = (beforePath: string, afterPath: string) => Promise<ScreenshotDiffMeasurement>;
/** A PNG's pixel size, from its header. */
async function pngSize(path: string): Promise<{ width: number; height: number }> {
const file = await open(path, "r");
try {
const header = Buffer.alloc(24);
await file.read(header, 0, 24, 0);
return { width: header.readUInt32BE(16), height: header.readUInt32BE(20) };
} finally { await file.close(); }
}
/**
* Compares two runs of the same flows file checkpoint by checkpoint, with the review's screenshot diff and the
* status-bar and home-indicator regions it sets aside. `measure` is only called for pairs with a screenshot on both
* sides of the same size.
*/
export async function compareFlowRuns(before: FlowRun, after: FlowRun, measure: MeasureScreenshots, threshold = 0.1): Promise<FlowComparison> {
const was = runConditions(before.report), now = runConditions(after.report);
const notes: string[] = [];
// A Duo's status bar sits on whichever panel was in use, and a run does not record the pose.
const foldable = isFoldable(before.report.exploration.app.platform, was.device);
if (foldable) notes.push("The status bar is not set aside: the run does not record which panel of the foldable was in use.");
const system = foldable ? [] : systemRegions({ platform: before.report.exploration.app.platform, device: was.device });
const checkpoints: ComparedCheckpoint[] = [];
for (const pair of pairCheckpoints(before.report, after.report)) {
if (!pair.before) { checkpoints.push({ ...pair, verdict: "added" }); continue; }
if (!pair.after) { checkpoints.push({ ...pair, verdict: "removed" }); continue; }
if (!("screenshot" in pair.before) || !("screenshot" in pair.after)) { checkpoints.push({ ...pair, verdict: "missing" }); continue; }
const beforePath = resolve(before.directory, pair.before.screenshot), afterPath = resolve(after.directory, pair.after.screenshot);
const [a, b] = await Promise.all([pngSize(beforePath), pngSize(afterPath)]);
if (a.width !== b.width || a.height !== b.height) { checkpoints.push({ ...pair, verdict: "different-size" }); continue; }
const raw = await measure(beforePath, afterPath);
const { measurement } = ignoreSystemRegions(raw, system, systemBandFrameLoader(raw, threshold));
const different = measurement.regions.reduce((sum, region) => sum + region.differentPixels, 0);
const mismatchPercentage = raw.totalPixels ? Math.round((different / raw.totalPixels) * 10000) / 100 : 0;
// As in a review, a mismatch that rounds to 0.00% is rendering noise.
const changed = measurement.regions.length > 0 && mismatchPercentage > 0;
checkpoints.push({ ...pair, verdict: changed ? "changed" : "unchanged", mismatchPercentage,
...(changed ? { regions: measurement.regions.map(region => region.normalizedRect) } : {}) });
}
const counts = { unchanged: 0, changed: 0, "different-size": 0, missing: 0, added: 0, removed: 0 };
for (const item of checkpoints) counts[item.verdict] += 1;
return { kind: FLOW_COMPARISON_KIND, before: { ...was, report: before.directory }, after: { ...now, report: after.directory },
conditions: conditionDifferences(was, now), notes, counts, checkpoints };
}
+83
View File
@@ -0,0 +1,83 @@
import { copyFile, mkdir, readFile, writeFile } from "node:fs/promises";
import { basename, dirname, join, resolve } from "node:path";
import type { CheckpointSide, FlowReview, FlowReviewRun, FlowReviewSide } from "../contracts/flow-comparison.js";
import type { ExplorationFlow, ExplorationReport } from "../contracts/standalone-report.js";
import type { FlowComparison, FlowRun } from "./flow-comparison.js";
const keyOf = (name: string) => name.trim().toLowerCase();
/**
* The text a screen showed, in tree order and once each: every label and value its accessibility tree kept. A run's
* artifacts sit beside its report directory; a report copied on its own has none, and its checkpoints carry no text.
*/
async function screenText(run: FlowRun, observationId: string): Promise<string[] | undefined> {
const observation = run.report.map.observations.find(item => item.id === observationId);
const artifact = observation?.accessibilityArtifactId ? run.report.map.artifacts.find(item => item.id === observation.accessibilityArtifactId) : undefined;
if (!artifact) return undefined;
const roots = basename(run.directory) === "report" ? [dirname(run.directory), run.directory] : [run.directory];
for (const root of roots) {
let tree: { nodes?: Array<{ label?: unknown; value?: unknown }> };
try { tree = JSON.parse(await readFile(resolve(root, artifact.path), "utf8")) as typeof tree; } catch { continue; }
const text = new Set<string>();
for (const node of tree.nodes ?? []) for (const value of [node.label, node.value]) {
if (typeof value === "string" && value.trim()) text.add(value.trim());
}
return [...text];
}
return undefined;
}
async function reviewSide(run: FlowRun, side: CheckpointSide | undefined): Promise<FlowReviewSide | undefined> {
if (!side) return undefined;
if ("missing" in side) return { missing: side.missing };
const text = await screenText(run, side.observationId);
return { screenshot: side.screenshot, ...(text ? { text } : {}) };
}
function reviewRun(report: ExplorationReport, name: string): FlowReviewRun | undefined {
const flow: ExplorationFlow | undefined = report.exploration.flows?.find(item => keyOf(item.name) === keyOf(name));
if (!flow) return undefined;
return { status: flow.status, ...(flow.reason ? { reason: flow.reason } : {}), ...(flow.steps ? { steps: flow.steps } : {}),
...(flow.failedStep !== undefined ? { failedStep: flow.failedStep } : {}) };
}
/** The comparison as the Viewer reads it: flows in the comparison's order, each with both runs' outcome and steps. */
export async function flowReview(comparison: FlowComparison, before: FlowRun, after: FlowRun): Promise<FlowReview> {
const flows: FlowReview["flows"] = [];
for (const item of comparison.checkpoints) {
let flow = flows.at(-1);
if (!flow || flow.name !== item.flow) {
const was = reviewRun(before.report, item.flow), now = reviewRun(after.report, item.flow);
flows.push(flow = { name: item.flow, ...(was ? { before: was } : {}), ...(now ? { after: now } : {}), checkpoints: [] });
}
const [was, now] = await Promise.all([reviewSide(before, item.before), reviewSide(after, item.after)]);
flow.checkpoints.push({ name: item.checkpoint, verdict: item.verdict, ...(was ? { before: was } : {}), ...(now ? { after: now } : {}),
...(item.regions ? { regions: item.regions } : {}) });
}
const { report: _before, ...was } = comparison.before, { report: _after, ...now } = comparison.after;
return { before: was, after: now, conditions: comparison.conditions, notes: comparison.notes, flows };
}
/**
* Writes a report the Viewer opens: the after run's exploration report with the review beside it, and every asset it
* names, the before run's checkpoint screenshots included. Assets are named by their SHA-256, so the two runs share one
* folder without clashing.
*/
export async function writeFlowReviewReport(comparison: FlowComparison, before: FlowRun, after: FlowRun, directory: string): Promise<string> {
const target = resolve(directory);
if ([before.directory, after.directory].some(source => resolve(source) === target)) throw new Error("The review report needs its own folder, not one of the runs' reports.");
const review = await flowReview(comparison, before, after);
await mkdir(join(target, "assets"), { recursive: true });
const assets = [...after.report.assets];
for (const asset of after.report.assets) await copyFile(resolve(after.directory, asset.path), join(target, asset.path));
const known = new Set(assets.map(asset => asset.path));
for (const asset of before.report.assets) {
const used = comparison.checkpoints.some(item => item.before && "screenshot" in item.before && item.before.screenshot === asset.path);
if (!used || known.has(asset.path)) continue;
await copyFile(resolve(before.directory, asset.path), join(target, asset.path));
assets.push(asset); known.add(asset.path);
}
const report: ExplorationReport = { ...after.report, assets, flowReview: review };
await writeFile(join(target, "exploration.json"), JSON.stringify(report));
return target;
}
+135
View File
@@ -0,0 +1,135 @@
import { createReadStream } from "node:fs";
import { lstat, open } from "node:fs/promises";
import { basename } from "node:path";
import { createGunzip } from "node:zlib";
import { Reader, Uint8ArrayWriter, ZipReader, configure } from "@zip.js/zip.js";
import { CiError } from "./ci-errors.js";
/**
* Reads the entries of tar.gz and zip archives without unpacking them: a prebuilt app, a downloaded controller, a
* baseline artifact. Nothing here writes a file; callers place entries themselves, through safeRelativePath.
*/
export const MAX_METADATA_BYTES = 4 * 1024 * 1024;
export const MAX_ARCHIVE_ENTRIES = 200_000;
export function archiveError(code: string, message: string, nextAction: string, details?: Record<string, unknown>): CiError {
return new CiError({ code, stage: "archive", message, nextAction, ...(details ? { details } : {}) });
}
// ---- tar.gz -------------------------------------------------------------------------------------
type TarVisitor = (entry: { name: string; type: string; size: number; mode: number }, contents: () => Promise<Buffer>, skip: () => Promise<void>) => Promise<boolean | void>;
/** Walk a gzip tarball once; `visit` returns true to stop. Long GNU names are honoured; nothing is written. */
export async function walkTar(path: string, visit: TarVisitor): Promise<void> {
const stream = createReadStream(path).pipe(createGunzip());
const iterator = stream[Symbol.asyncIterator]();
let buffer = Buffer.alloc(0);
let done = false;
const fill = async (needed: number) => {
while (buffer.length < needed && !done) {
const next = await iterator.next();
if (next.done) { done = true; break; }
buffer = Buffer.concat([buffer, next.value as Buffer]);
}
};
const take = async (count: number): Promise<Buffer> => {
await fill(count);
if (buffer.length < count) throw archiveError("ARCHIVE_INVALID", "The tarball ended unexpectedly.", "Pass a complete .tar.gz archive.");
const out = buffer.subarray(0, count); buffer = buffer.subarray(count); return Buffer.from(out);
};
const drop = async (count: number) => {
let remaining = count;
while (remaining > 0) {
await fill(Math.min(remaining, 1 << 20));
const step = Math.min(remaining, buffer.length);
if (!step) throw archiveError("ARCHIVE_INVALID", "The tarball ended unexpectedly.", "Pass a complete .tar.gz archive.");
buffer = buffer.subarray(step); remaining -= step;
}
};
let longName: string | undefined;
let entries = 0;
try {
for (;;) {
await fill(512);
if (buffer.length < 512 || buffer.subarray(0, 512).every(byte => byte === 0)) return;
const header = await take(512);
const field = (offset: number, length: number) => header.subarray(offset, offset + length).toString("utf8").replace(/\0.*$/s, "");
const size = parseInt(field(124, 12).trim() || "0", 8);
const type = field(156, 1) || "0";
const prefix = field(345, 155);
const name = longName ?? (prefix ? `${prefix}/${field(0, 100)}` : field(0, 100));
longName = undefined;
const padded = Math.ceil(size / 512) * 512;
if (type === "L") { longName = (await take(padded)).subarray(0, size).toString("utf8").replace(/\0+$/, ""); continue; }
if (++entries > MAX_ARCHIVE_ENTRIES) throw archiveError("ARCHIVE_TOO_LARGE", "The tarball has too many entries.", "Pass a smaller archive.");
let consumed = false;
const stop = await visit({ name, type, size, mode: parseInt(field(100, 8).trim() || "0", 8) },
async () => { consumed = true; return (await take(padded)).subarray(0, size); },
async () => { consumed = true; await drop(padded); });
if (!consumed) await drop(padded);
if (stop) return;
}
} finally { await iterator.return?.(undefined); }
}
export function safeRelativePath(name: string): string | undefined {
const parts = name.replace(/^\.\//, "").split("/").filter(part => part.length);
if (!parts.length || parts.some(part => part === "." || part === ".." || part.includes("\\") || part.includes("\0"))) return undefined;
return parts.join("/");
}
// ---- zip ----------------------------------------------------------------------------------------
export type ZipEntry = Awaited<ReturnType<ZipReader<unknown>["getEntries"]>>[number];
/** zip.js reads the central directory and entries by range, so a 200 MB APK is never loaded whole. */
class FileRangeReader extends Reader<unknown> {
constructor(private readonly path: string) { super(path); }
override async init(): Promise<void> {
this.size = (await lstat(this.path)).size;
}
/** Each range opens its own descriptor: nothing stays open after the entries are read, so no handle leaks when a caller stops early. */
override async readUint8Array(index: number, length: number): Promise<Uint8Array> {
const out = new Uint8Array(length);
const handle = await open(this.path, "r");
try {
let offset = 0;
while (offset < length) {
const { bytesRead } = await handle.read(out, offset, length - offset, index + offset);
if (bytesRead === 0) break;
offset += bytesRead;
}
return offset === length ? out : out.subarray(0, offset);
} finally { await handle.close(); }
}
}
const readers = new WeakMap<object, FileRangeReader>();
/** The ABIs an APK ships native libraries for (`lib/<abi>/`), without reading its manifest. */
export async function apkAbis(path: string): Promise<string[]> {
return [...new Set((await zipEntries(path)).map(entry => /^lib\/([^/]+)\//.exec(entry.filename)?.[1]).filter((abi): abi is string => !!abi))].sort();
}
export async function zipEntries(path: string): Promise<ZipEntry[]> {
configure({ useWebWorkers: false });
const reader = new FileRangeReader(path);
const zip = new ZipReader(reader, { useWebWorkers: false });
try {
const entries = await zip.getEntries();
if (entries.length > MAX_ARCHIVE_ENTRIES) throw archiveError("ARCHIVE_TOO_LARGE", "The archive has too many entries.", "Pass a smaller archive.");
for (const entry of entries) readers.set(entry, reader);
return entries;
} catch (error) {
if (error instanceof CiError) throw error;
throw archiveError("ARCHIVE_INVALID", `${basename(path)} is not a readable ZIP archive: ${error instanceof Error ? error.message : String(error)}`, "Pass a valid .ipa or .apk.");
}
}
export async function readZipEntry(entry: ZipEntry, maxBytes = MAX_METADATA_BYTES): Promise<Uint8Array> {
if (entry.directory || !entry.getData) throw archiveError("ARCHIVE_INVALID", `${entry.filename} is not a file entry.`, "Pass a valid archive.");
if (entry.uncompressedSize > maxBytes) throw archiveError("ARCHIVE_INVALID", `${entry.filename} is too large.`, "Pass a valid archive.");
return entry.getData(new Uint8ArrayWriter(), { checkSignature: true, useWebWorkers: false });
}
+1 -1
View File
@@ -11,7 +11,7 @@ import { chmod, mkdir, open, readdir, rename, rm, stat, writeFile } from "node:f
import { homedir } from "node:os";
import { dirname, join, resolve, sep } from "node:path";
import { fileURLToPath } from "node:url";
import { safeRelativePath, walkTar } from "../binary/archive.js";
import { safeRelativePath, walkTar } from "./archive-entries.js";
import { CiError } from "./ci-errors.js";
import { AGENT_DEVICE_VERSION } from "./agent-device-version.js";
import { ARGENT_VERSION } from "./argent-version.js";
+1 -1
View File
@@ -2,7 +2,7 @@
// instance), and every tsImport call loads its own copy of each module.
export { runBlackBoxExploration } from "../../src/binary/explorer.js";
export { assertFlowsFit, createFlowSession, flowsNeedModel, parseFlowPlan } from "../../src/binary/flows.js";
export { hintInputs } from "../../src/binary/run.js";
export { chooseStrategy, startStrategy, strategyInputs } from "../../src/binary/strategy.js";
export { runBoundedSession } from "../../src/capture/agent/session-shared.js";
export { ExploreStore, validateMap } from "../../src/map/store.js";
export { CiError } from "../../src/shared/ci-errors.js";
+7 -5
View File
@@ -9,7 +9,8 @@ const { runBlackBoxExploration, screenEvidenceKey, elementsOf, fieldName, system
const { ExploreStore, validateMap } = await tsImport("../../src/map/store.ts", import.meta.url);
const { CONTRACT } = await tsImport("../../src/types.ts", import.meta.url);
const tsRun = await tsImport("../../src/binary/run.ts", import.meta.url);
const { assertAppUrl, describeForeground, foreignNode, guardPositionals, leftForeground, retryWhileBusy } = tsRun;
const { describeForeground, foreignNode, guardPositionals, leftForeground, retryWhileBusy } = tsRun;
const { assertAppUrl } = await tsImport("../../src/binary/app-url.ts", import.meta.url);
const PNG = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==", "base64");
const SCREENS = {
@@ -246,7 +247,8 @@ test("a {{secret:NAME}} from the hints is typed on the device, while the model,
});
test("hint inputs are read before a session starts and name what to set", async () => {
const { hintInputs } = await tsImport("../../src/binary/run.ts", import.meta.url);
const { strategyInputs } = await tsImport("../../src/binary/strategy.ts", import.meta.url);
const hintInputs = (hints, environment) => strategyInputs({ kind: "model" }, hints, environment);
assert.deepEqual(hintInputs("Explore the settings", {}).names, []);
assert.deepEqual(hintInputs("Use {{secret:APP_TEST_EMAIL}} and {{secret:APP_TEST_PASSWORD}}", { APP_TEST_EMAIL: "[email protected]", APP_TEST_PASSWORD: "pw" }).names, ["APP_TEST_EMAIL", "APP_TEST_PASSWORD"]);
assert.throws(() => hintInputs("Use {{secret:APP_TEST_PASSWORD}}", {}), error => error.code === "EXPLORATION_SECRET_MISSING" && /export APP_TEST_PASSWORD=/.test(error.nextAction));
@@ -854,7 +856,7 @@ test("a coordinate tap with no description on an icon-font button is named by wh
});
test("with no --provider, model access falls through API keys and local logins to the first that passes preflight", async () => {
const { preflightModelSession } = await tsImport("../../src/binary/run.ts", import.meta.url);
const { preflightModelSession } = await tsImport("../../src/binary/strategy.ts", import.meta.url);
const tried = [];
const working = "anthropic/local-login";
const createSession = (config, { auth }) => ({ async preflight() {
@@ -870,7 +872,7 @@ test("with no --provider, model access falls through API keys and local logins t
});
test("an explicit --provider is the only model access tried, and its preflight error is the one reported", async () => {
const { preflightModelSession } = await tsImport("../../src/binary/run.ts", import.meta.url);
const { preflightModelSession } = await tsImport("../../src/binary/strategy.ts", import.meta.url);
const tried = [];
const createSession = (config, { auth }) => ({ async preflight() { tried.push(`${config.provider}/${auth}`); throw new Error("refused"); } });
await assert.rejects(preflightModelSession({ provider: "openai", createSession }, { ANTHROPIC_API_KEY: "key" }, () => {}), /refused/);
@@ -878,7 +880,7 @@ test("an explicit --provider is the only model access tried, and its preflight e
});
test("when nothing passes preflight, the first failure is reported", async () => {
const { preflightModelSession } = await tsImport("../../src/binary/run.ts", import.meta.url);
const { preflightModelSession } = await tsImport("../../src/binary/strategy.ts", import.meta.url);
const createSession = (config, { auth }) => ({ async preflight() { throw new Error(`${config.provider} ${auth} refused`); } });
await assert.rejects(preflightModelSession({ createSession }, {}, () => {}), /anthropic local-login refused/);
});
+40 -6
View File
@@ -5,7 +5,7 @@ import { join } from "node:path";
import test from "node:test";
import { tsImport } from "tsx/esm/api";
const { runBlackBoxExploration, assertFlowsFit, createFlowSession, flowsNeedModel, parseFlowPlan, hintInputs, runBoundedSession, ExploreStore, validateMap, CiError, CONTRACT } =
const { runBlackBoxExploration, assertFlowsFit, createFlowSession, flowsNeedModel, parseFlowPlan, chooseStrategy, startStrategy, strategyInputs, runBoundedSession, ExploreStore, validateMap, CiError, CONTRACT } =
await tsImport("../helpers/flows-host.ts", import.meta.url);
const PNG = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==", "base64");
@@ -113,6 +113,19 @@ test("a failed step ends its flow with the reason and the checkpoints it did not
assert.deepEqual(await validateMap(map, store), []);
});
test("the session records each flow's steps as the flows file names them, and the step a failed flow stopped at", async () => {
const plan = parseFlowPlan({ flows: [
{ name: "Broken", steps: [{ checkpoint: "Home" }, { action: "tap", label: "Nowhere" }, { checkpoint: "Never" }] },
{ name: "Settings", steps: [{ action: "tap", label: "Settings" }, { checkpoint: "Settings" }] },
] });
const session = createFlowSession(plan, "ios", { app: { name: "Example" } });
await explore(plan, { session });
assert.deepEqual(session.steps(), [
{ flow: "Broken", steps: [{ checkpoint: "Home" }, { action: 'tap "Nowhere"' }, { checkpoint: "Never" }], failedStep: 2 },
{ flow: "Settings", steps: [{ action: 'tap "Settings"' }, { checkpoint: "Settings" }] },
]);
});
test("a destructive tap in a flow is refused by the explorer, which fails the flow", async () => {
const { result, acted } = await explore(parseFlowPlan({ flows: [{ name: "Danger", steps: [{ action: "tap", label: "Delete account" }, { checkpoint: "After" }] }] }));
assert.ok(!acted.some(action => action.label === "Delete account"));
@@ -185,8 +198,21 @@ test("a flows file is checked before any device starts", () => {
test("a {{secret:NAME}} in a described step must be set before any device starts, as one in the hints must", () => {
const plan = parseFlowPlan({ flows: [{ name: "Sign in", steps: [{ do: "Sign in with {{secret:TEST_PASSWORD}}" }, { checkpoint: "Home" }] }] });
assert.throws(() => hintInputs(undefined, {}, undefined, plan), error => error.code === "EXPLORATION_SECRET_MISSING" && /TEST_PASSWORD/.test(error.nextAction));
assert.deepEqual(hintInputs(undefined, { TEST_PASSWORD: "hunter2" }, undefined, plan).names, ["TEST_PASSWORD"]);
assert.throws(() => strategyInputs({ kind: "flows", plan }, undefined, {}), error => error.code === "EXPLORATION_SECRET_MISSING" && /TEST_PASSWORD/.test(error.nextAction));
assert.deepEqual(strategyInputs({ kind: "flows", plan }, undefined, { TEST_PASSWORD: "hunter2" }).names, ["TEST_PASSWORD"]);
});
test("the strategy is chosen before any device starts, and options that would do nothing with it are refused", () => {
const exact = { flows: [{ name: "A", steps: [{ action: "tap", label: "Settings" }, { checkpoint: "Settings" }] }] };
const described = { flows: [{ name: "A", steps: [{ do: "Open the settings" }, { checkpoint: "Settings" }] }] };
assert.deepEqual(chooseStrategy({}), { kind: "model" });
assert.equal(chooseStrategy({ flows: exact }).kind, "flows");
assert.equal(chooseStrategy({ flows: described, provider: "openai", hints: "Use the test account" }).kind, "flows");
assert.throws(() => chooseStrategy({ flows: exact, replay: { steps: [{ capture: "Home" }] } }), { code: "FLOWS_WITH_REPLAY" });
assert.throws(() => chooseStrategy({ flows: exact, model: "gpt-6.1-sol" }), error => error.code === "FLOWS_OPTIONS_IGNORED" && /--model would do nothing/.test(error.message));
assert.throws(() => chooseStrategy({ replay: { steps: [{ capture: "Home" }] }, hints: "x" }), { code: "REPLAY_OPTIONS_IGNORED" });
assert.throws(() => chooseStrategy({ flows: { flows: [{ name: "A", steps: [{ action: "open", url: "https://example.com" }, { checkpoint: "Web" }] }] } }),
error => error.code === "FLOWS_PLAN_INVALID" && /Flow 1 \(A\), step 1: https: URLs leave the app/.test(error.message));
});
test("a checkpoint after a relaunch takes a new picture, even of the screen the previous flow ended on untouched", async () => {
@@ -264,8 +290,8 @@ test("a described step limited to another platform needs no model, and its secre
assert.equal(flowsNeedModel(plan, "ios"), false);
assert.equal(flowsNeedModel(plan, "android"), true);
assert.doesNotThrow(() => createFlowSession(plan, "ios", { app: { name: "Example" } }));
assert.deepEqual(hintInputs(undefined, {}, undefined, plan, "ios").names, []);
assert.throws(() => hintInputs(undefined, {}, undefined, plan), error => error.code === "EXPLORATION_SECRET_MISSING", "without a platform, every step's secrets are needed");
assert.deepEqual(strategyInputs({ kind: "flows", plan }, undefined, {}, "ios").names, []);
assert.throws(() => strategyInputs({ kind: "flows", plan }, undefined, {}), error => error.code === "EXPLORATION_SECRET_MISSING", "without a platform, every step's secrets are needed");
});
test("flows whose tool calls cannot fit the turn budget are refused before any device starts", () => {
@@ -275,6 +301,14 @@ test("flows whose tool calls cannot fit the turn budget are refused before any d
assert.doesNotThrow(() => assertFlowsFit(flows(41), "ios", limits), "41 flows take 124 calls");
});
test("flows whose described steps run only on another platform start with no model at all", async () => {
const plan = parseFlowPlan({ flows: [{ name: "A", steps: [{ do: "Sign in", platforms: ["android"] }, { action: "tap", label: "Settings" }, { checkpoint: "Settings" }] }] });
const createSession = () => { throw new Error("no model may be started"); };
const strategy = await startStrategy({ kind: "flows", plan }, { platform: "ios", limits: { maxScreens: 10, maxActions: 10 }, options: { createSession }, environment: {}, say() {}, app: { name: "Example" } });
assert.deepEqual([strategy.provider, strategy.model, strategy.auth], ["flows", "flows file", "none"]);
assert.deepEqual(strategy.flows(), [{ flow: "A", checkpoints: ["Settings"] }]);
});
test("a checkpoint whose screen withholds its picture is recorded as missing, its flow goes on, and the summary names it", async () => {
const plan = parseFlowPlan({ flows: [{ name: "Home", steps: [{ checkpoint: "Home" }, { action: "wait", milliseconds: 1 }, { checkpoint: "Still home" }, { action: "tap", label: "Settings" }, { checkpoint: "Settings" }] }] });
const { result, map } = await explore(plan, { device: { withholdPicture: [2] } });
@@ -330,7 +364,7 @@ test("a flow with no checkpoint on the platform is skipped, and its steps need n
{ name: "Settings", steps: [{ action: "tap", label: "Settings" }, { checkpoint: "Settings" }] },
] });
assert.equal(flowsNeedModel(plan, "ios"), false);
assert.deepEqual(hintInputs(undefined, {}, undefined, plan, "ios").names, []);
assert.deepEqual(strategyInputs({ kind: "flows", plan }, undefined, {}, "ios").names, []);
const { result } = await explore(plan);
assert.deepEqual(result.flows.map(flow => flow.name), ["Settings"]);
assert.match(result.summary, /^Ran 1 flow on ios/);
+2 -1
View File
@@ -8,7 +8,8 @@ const transition = await tsImport(
);
test("canvas view direction follows the toolbar order", () => {
assert.deepEqual(transition.CANVAS_VIEW_ORDER, ["map", "screens", "compare"]);
assert.deepEqual(transition.CANVAS_VIEW_ORDER, ["map", "screens", "compare", "flows"]);
assert.equal(transition.canvasViewTransitionDirection("compare", "flows"), "forward");
assert.equal(transition.canvasViewTransitionDirection("screens", "compare"), "forward");
assert.equal(transition.canvasViewTransitionDirection("map", "screens"), "forward");
assert.equal(transition.canvasViewTransitionDirection("compare", "screens"), "backward");
+160
View File
@@ -0,0 +1,160 @@
import assert from "node:assert/strict";
import { access, mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";
import { createRequire } from "node:module";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
import { tsImport } from "tsx/esm/api";
const { pairCheckpoints, conditionDifferences, runConditions } = await tsImport("../../src/contracts/flow-comparison.ts", import.meta.url);
const { compareFlowRuns } = await tsImport("../../src/review/flow-comparison.ts", import.meta.url);
const { writeFlowReviewReport } = await tsImport("../../src/review/flow-review.ts", import.meta.url);
const { PNG } = createRequire(import.meta.url)("pngjs");
const png = (width, height) => PNG.sync.write(new PNG({ width, height }));
/**
* A minimal exploration report of a flows run, written to a directory. `flows` maps each flow name to its checkpoints,
* each `[name, size]` for a picture of that size or `[name, { missing }]` for one without.
*/
async function run(flows, options = {}) {
const directory = await mkdtemp(join(tmpdir(), "explore-flow-compare-"));
await mkdir(join(directory, "assets"));
const report = { kind: "explore.exploration-report", assets: [],
map: { run: { environment: { deviceModel: options.device ?? "iPhone 17 Pro", locale: options.locale ?? "en_US", orientation: "portrait", colorScheme: options.colorScheme ?? "light" } }, observations: [], artifacts: [] },
exploration: { app: { name: "Example", bundleId: options.bundleId ?? "dev.example.app", platform: options.platform ?? "ios" }, device: { model: options.device ?? "iPhone 17 Pro", osVersion: options.osVersion ?? "26.0" }, flows: [] } };
let index = 0;
for (const [name, checkpoints] of Object.entries(flows)) {
const flow = { name, status: "completed", checkpoints: [], ...options.flow?.[name] };
for (const [checkpoint, shot] of checkpoints) {
if (shot.missing) { flow.checkpoints.push({ name: checkpoint, missing: shot.missing }); continue; }
// Named by a digest, as a run names its assets: two runs never share a name for different pictures.
const sha = createHash("sha256").update(`${directory}:${++index}`).digest("hex"), id = `obs_${index}`;
await writeFile(join(directory, "assets", `${sha}.png`), png(...shot));
report.assets.push({ sha256: sha, path: `assets/${sha}.png` });
report.map.artifacts.push({ id: `art_${index}`, kind: "screenshot", sha256: sha });
const observation = { id, screenshotArtifactId: `art_${index}` };
const text = options.text?.[checkpoint];
if (text) {
// The accessibility tree a run keeps beside its report, as the explorer writes it.
const path = `artifacts/accessibility/${index}.json`;
await mkdir(join(directory, "artifacts", "accessibility"), { recursive: true });
await writeFile(join(directory, path), JSON.stringify({ nodes: text.map(label => ({ label })) }));
report.map.artifacts.push({ id: `acc_${index}`, kind: "accessibility", path });
observation.accessibilityArtifactId = `acc_${index}`;
}
report.map.observations.push(observation);
flow.checkpoints.push({ name: checkpoint, observationId: id });
}
report.exploration.flows.push(flow);
}
return { report, directory };
}
/** A diff that finds the given regions (in percent), each 1% of the pixels. */
const measuring = (regionsFor = () => []) => async (baselinePath, currentPath) => {
const regions = regionsFor(baselinePath, currentPath).map((normalizedRect, index) => ({ index, normalizedRect, rect: normalizedRect, differentPixels: 100, shareOfDiffPercentage: 0, densityPercentage: 0, shape: "", size: "" }));
return { baselinePath, currentPath, totalPixels: 10_000, differentPixels: regions.length * 100, mismatchPercentage: regions.length, match: regions.length === 0, regions };
};
test("checkpoints pair by flow and checkpoint name, ignoring case, in the after run's order", async () => {
const before = await run({ "Sign in": [["Welcome", [4, 8]], ["Home", [4, 8]]], Settings: [["Settings", [4, 8]]] });
const after = await run({ "sign in": [["home", [4, 8]], ["welcome", { missing: "The screenshot was blocked." }]], Profile: [["Profile", [4, 8]]] });
const pairs = pairCheckpoints(before.report, after.report);
assert.deepEqual(pairs.map(pair => [pair.flow, pair.checkpoint, !!pair.before, !!pair.after]), [
["sign in", "home", true, true], ["sign in", "welcome", true, true], ["Profile", "Profile", false, true], ["Settings", "Settings", true, false]]);
assert.deepEqual(pairs[1].after, { missing: "The screenshot was blocked." });
assert.equal(pairs[0].before.screenshot, before.report.assets[1].path, "Home is the before run's second picture");
});
test("a run's conditions that differ are listed, not refused", async () => {
const before = await run({ A: [["Home", [4, 8]]] }), after = await run({ A: [["Home", [4, 8]]] }, { osVersion: "27.0", colorScheme: "dark" });
assert.deepEqual(conditionDifferences(runConditions(before.report), runConditions(after.report)),
[{ field: "osVersion", before: "26.0", after: "27.0" }, { field: "colorScheme", before: "light", after: "dark" }]);
});
test("each pair gets a verdict: unchanged, changed with where, missing, different-size, added or removed", async () => {
const before = await run({ Flow: [["Same", [4, 8]], ["Moved", [4, 8]], ["Gone", [4, 8]], ["Blocked", [4, 8]], ["Rotated", [4, 8]]] });
const after = await run({ Flow: [["Same", [4, 8]], ["Moved", [4, 8]], ["New", [4, 8]], ["Blocked", { missing: "A secret was on screen." }], ["Rotated", [8, 4]]] });
const measured = [];
const result = await compareFlowRuns(before, after, measuring((baseline, current) => {
measured.push([baseline, current]);
return baseline.endsWith(before.report.assets[1].path) ? [{ x: 10, y: 40, width: 20, height: 10 }] : [];
}));
assert.deepEqual(result.checkpoints.map(item => [item.checkpoint, item.verdict]),
[["Same", "unchanged"], ["Moved", "changed"], ["New", "added"], ["Blocked", "missing"], ["Rotated", "different-size"], ["Gone", "removed"]]);
assert.equal(measured.length, 2, "only pairs with a picture of the same size on both sides are diffed");
assert.deepEqual(result.checkpoints[1].regions, [{ x: 10, y: 40, width: 20, height: 10 }]);
assert.equal(result.checkpoints[1].mismatchPercentage, 1);
assert.deepEqual(result.counts, { unchanged: 1, changed: 1, "different-size": 1, missing: 1, added: 1, removed: 1 });
assert.deepEqual(result.conditions, []);
});
test("a difference only in the status bar is set aside, as in a review", async () => {
const before = await run({ Flow: [["Home", [4, 8]]] }), after = await run({ Flow: [["Home", [4, 8]]] });
const result = await compareFlowRuns(before, after, measuring(() => [{ x: 80, y: 1, width: 10, height: 2 }]));
assert.equal(result.checkpoints[0].verdict, "unchanged");
});
test("on a foldable, the status bar is not set aside and the result says why", async () => {
const before = await run({ Flow: [["Home", [4, 8]]] }, { device: "iPhone Duo" }), after = await run({ Flow: [["Home", [4, 8]]] }, { device: "iPhone Duo" });
// Where the folded panel's status bar is: guessing that panel would hide a change the open panel shows there.
const result = await compareFlowRuns(before, after, measuring(() => [{ x: 86, y: 13, width: 4, height: 4 }]));
assert.equal(result.checkpoints[0].verdict, "changed");
assert.match(result.notes[0], /foldable/);
});
test("compare:flows reads two output directories, needs no diff when no pair has two pictures, and refuses a run without flows", async () => {
const { spawnSync } = await import("node:child_process");
const root = join(import.meta.dirname, "../..");
const cli = join(root, "dist", "cli.js");
const before = await run({ Flow: [["Home", { missing: "Not reached." }]] }), after = await run({ Flow: [["Home", [4, 8]]] });
// explore:binary leaves the report in <output>/report.
const output = await mkdtemp(join(tmpdir(), "explore-flow-compare-out-"));
await mkdir(join(output, "report", "assets"), { recursive: true });
await writeFile(join(output, "report", "exploration.json"), JSON.stringify(after.report));
await writeFile(join(before.directory, "exploration.json"), JSON.stringify(before.report));
const compared = spawnSync(process.execPath, [cli, "compare:flows", "--before", before.directory, "--after", output], { cwd: root, encoding: "utf8" });
assert.equal(compared.status, 0, compared.stderr);
const result = JSON.parse(compared.stdout);
assert.deepEqual(result.comparison.checkpoints.map(item => [item.checkpoint, item.verdict]), [["Home", "missing"]]);
const plain = await run({});
delete plain.report.exploration.flows;
await writeFile(join(plain.directory, "exploration.json"), JSON.stringify(plain.report));
const refused = spawnSync(process.execPath, [cli, "compare:flows", "--before", plain.directory, "--after", output], { cwd: root, encoding: "utf8" });
assert.notEqual(refused.status, 0);
assert.match(JSON.parse(refused.stderr).error, /did not follow a flows file/);
});
test("a review report is the after run's report with the review beside it: both builds' pictures and text, each flow's outcome and steps", async () => {
const steps = [{ checkpoint: "Home" }, { action: 'tap "Continue"' }, { checkpoint: "Done" }];
const before = await run({ Flow: [["Home", [4, 8]], ["Done", [4, 8]]] }, { text: { Home: ["Welcome", "Sign in"], Done: ["All set"] }, flow: { Flow: { steps } } });
const after = await run({ Flow: [["Home", [4, 8]], ["Done", { missing: "The flow did not reach it." }]] },
{ text: { Home: ["Welcome", "Log in"] }, flow: { Flow: { status: "failed", reason: "Step 2 (tap \"Continue\") failed: no such element", steps, failedStep: 2 } } });
const comparison = await compareFlowRuns(before, after, measuring(() => [{ x: 10, y: 40, width: 20, height: 10 }]));
const target = await mkdtemp(join(tmpdir(), "explore-flow-review-"));
await writeFlowReviewReport(comparison, before, after, target);
const written = JSON.parse(await readFile(join(target, "exploration.json"), "utf8"));
assert.equal(written.kind, "explore.exploration-report");
assert.deepEqual(written.exploration, after.report.exploration, "the after run's report is kept as it was");
const [flow] = written.flowReview.flows;
assert.deepEqual(flow.after, { status: "failed", reason: 'Step 2 (tap "Continue") failed: no such element', steps, failedStep: 2 });
assert.deepEqual(flow.before, { status: "completed", steps });
assert.deepEqual(flow.checkpoints.map(item => [item.name, item.verdict]), [["Home", "changed"], ["Done", "missing"]]);
assert.deepEqual(flow.checkpoints[0].before.text, ["Welcome", "Sign in"]);
assert.deepEqual(flow.checkpoints[0].after.text, ["Welcome", "Log in"]);
assert.deepEqual(flow.checkpoints[0].regions, [{ x: 10, y: 40, width: 20, height: 10 }]);
assert.deepEqual(flow.checkpoints[1].after, { missing: "The flow did not reach it." });
// Every picture the review names is in the report's folder, the before run's included.
for (const checkpoint of flow.checkpoints) for (const side of [checkpoint.before, checkpoint.after]) {
if (side?.screenshot) await access(join(target, side.screenshot));
}
assert.equal(written.assets.length, 3);
});
test("a review report is not written over one of the runs' own reports", async () => {
const before = await run({ Flow: [["Home", [4, 8]]] }), after = await run({ Flow: [["Home", [4, 8]]] });
const comparison = await compareFlowRuns(before, after, measuring());
await assert.rejects(writeFlowReviewReport(comparison, before, after, after.directory), /its own folder/);
});
+82
View File
@@ -0,0 +1,82 @@
import assert from "node:assert/strict";
import test from "node:test";
import { tsImport } from "tsx/esm/api";
const model = await tsImport("../../viewer/src/utils/flows-model.ts", import.meta.url);
const shot = (sha, text) => ({ screenshot: `assets/${sha.padStart(64, "0")}.png`, ...(text ? { text } : {}) });
const steps = [{ action: 'tap "Settings"' }, { checkpoint: "Settings" }, { do: "Open the output picker" }, { checkpoint: "Output" }, { do: "Close it" }, { do: "Open the input picker" }, { checkpoint: "Input" }];
const completed = { name: "Connect", before: { status: "completed", steps }, after: { status: "completed", steps }, checkpoints: [
{ name: "Settings", verdict: "unchanged", before: shot("1"), after: shot("2") },
{ name: "Output", verdict: "changed", before: shot("3", ["Output", "No outputs over USB-C."]), after: shot("4", ["Output", "No outputs over USB-C or Bluetooth."]), regions: [{ x: 1, y: 2, width: 3, height: 4 }] },
{ name: "Input", verdict: "unchanged", before: shot("5"), after: shot("6") },
] };
const stopped = { name: "Share", before: { status: "completed", steps }, after: { status: "failed", reason: "The model used its 40 tool calls without finishing the step.", steps, failedStep: 3 }, checkpoints: [
{ name: "Settings", verdict: "changed", before: shot("7"), after: shot("8") },
{ name: "Output", verdict: "missing", before: shot("9"), after: { missing: "The flow did not reach it." } },
{ name: "Input", verdict: "missing", before: shot("a"), after: { missing: "The flow did not reach it." } },
] };
const quiet = { name: "Edit", before: { status: "completed" }, after: { status: "completed" }, checkpoints: [{ name: "Home", verdict: "unchanged", before: shot("b"), after: shot("c") }] };
const review = { before: {}, after: {}, conditions: [], notes: [], flows: [completed, stopped, quiet] };
test("Changed keeps every flow with a checkpoint that is not unchanged, and every run that stopped; All keeps every flow", () => {
assert.deepEqual(model.visibleFlows(review, "changed").map(flow => flow.name), ["Connect", "Share"]);
assert.deepEqual(model.visibleFlows(review, "all").map(flow => flow.name), ["Connect", "Share", "Edit"]);
const stoppedWithoutChanges = { ...quiet, after: { status: "stopped" } };
assert.equal(model.flowNeedsReview(stoppedWithoutChanges), true, "a stopped run is never hidden by the filter");
});
test("a flow's mark and status line say whether it stopped, then how many checkpoints of each kind", () => {
assert.equal(model.flowMark(completed), "changed");
assert.equal(model.flowStatusLine(completed), "1 changed");
assert.equal(model.flowMark(stopped), "stopped");
assert.equal(model.flowStatusLine(stopped), "Stopped · 2 not reached · 1 changed");
assert.equal(model.flowMark(quiet), "unchanged");
assert.equal(model.flowStatusLine(quiet), "Unchanged");
assert.equal(model.flowMark({ ...quiet, before: undefined }), "added");
assert.equal(model.flowMark({ ...quiet, after: undefined }), "removed");
});
test("a flow opens at its first problem", () => {
assert.equal(model.firstProblem(completed), 1);
assert.deepEqual(model.problemIndexes(stopped), [0, 1, 2]);
assert.equal(model.firstProblem(quiet), 0, "a flow with nothing changed opens at its first checkpoint");
});
test("a missing picture says which build has none", () => {
assert.equal(model.checkpointLabel(stopped.checkpoints[1]), "Not reached");
const reachedOnlyAfter = { name: "Output", verdict: "missing", before: { missing: "The flow did not reach it." }, after: shot("d") };
assert.equal(model.checkpointLabel(reachedOnlyAfter), "Not reached before");
assert.equal(model.checkpointMark({ name: "Rotated", verdict: "different-size" }), "not-comparable");
});
test("the steps before a checkpoint are the ones since the previous checkpoint, with the failed one and those never run marked", () => {
assert.deepEqual(model.stepsBefore(completed, 0), [{ step: { action: 'tap "Settings"' }, failed: false, ran: true }]);
assert.deepEqual(model.stepsBefore(completed, 2).map(item => item.step), [{ do: "Close it" }, { do: "Open the input picker" }]);
assert.deepEqual(model.stepsBefore(stopped, 1), [{ step: { do: "Open the output picker" }, failed: true, ran: true }]);
assert.deepEqual(model.stepsBefore(stopped, 2).map(item => item.ran), [false, false]);
assert.deepEqual(model.stepsBefore(quiet, 0), [], "a run from before steps were recorded has none to show");
});
test("text changes are the lines one build's screen showed and the other's did not", () => {
assert.deepEqual(model.textChanges(completed.checkpoints[1]), { removed: ["No outputs over USB-C."], added: ["No outputs over USB-C or Bluetooth."] });
assert.equal(model.textChanges(completed.checkpoints[0]), undefined, "without the text of both screens there is nothing to compare");
assert.equal(model.textChanges(stopped.checkpoints[1]), undefined);
});
test("a picture is found by the SHA-256 its asset is named with", () => {
assert.deepEqual(model.sidePicture(shot("4")), { sha256: "4".padStart(64, "0") });
assert.deepEqual(model.sidePicture({ missing: "Blocked." }), { missing: "Blocked." });
assert.equal(model.sidePicture(undefined), undefined);
});
test("a text-sized change is spelled out in one line; a larger one is read off the picture", () => {
const changed = (before, after, height = 2) => ({ name: "Output", verdict: "changed", before: shot("3", before), after: shot("4", after), regions: [{ x: 0, y: 40, width: 50, height }] });
assert.equal(model.textChangeLine(changed(["Output", "No outputs."], ["Output", "No outputs over USB-C."])), "“No outputs over USB-C.”, was “No outputs.”");
assert.equal(model.textChangeLine(changed(["A"], ["A", "B", "C"])), "Adds “B” and 1 more");
assert.equal(model.textChangeLine(changed(["A", "B"], ["A"])), "Drops “B”");
assert.equal(model.textChangeLine(changed(["A"], ["B"], 12)), undefined, "a change taller than a line or two of text");
assert.equal(model.textChangeLine(changed(["A"], ["A"])), undefined, "the same text, so the change is not in it");
assert.equal(model.textChangeLine(stopped.checkpoints[0]), undefined, "without the text of both screens there is nothing to say");
});
+9 -1
View File
@@ -193,10 +193,18 @@ test("a screen without a route, or one the map does not know yet, travels as an
assert.equal(nav.formatRoute(pending, ctx), "/screens/(tabs)/later");
});
test("a flow review opens on Flows, and a link that names another view still opens it", () => {
const flows = { ...ctx, flows: true };
assert.equal(nav.parseLocation({ pathname: "/", search: "", hash: "" }, flows).canvas, "flows");
assert.equal(nav.parseLocation({ pathname: "/flows", search: "", hash: "" }, flows).canvas, "flows");
assert.equal(nav.parseLocation({ pathname: "/screens", search: "", hash: "" }, flows).canvas, "screens");
assert.equal(nav.formatRoute(nav.parseLocation({ pathname: "/flows", search: "", hash: "" }, flows), flows), "/flows");
});
test("legacy hashes still parse", () => {
assert.deepEqual(parse("/#workspace"), base);
assert.deepEqual(parse("/#workspace/screens"), { ...base, canvas: "screens" });
// Flows was retired; its links land on the map.
// Flows is a view only of a flow review: elsewhere its links, the retired Flows canvas's among them, land on the map.
assert.deepEqual(parse("/#workspace/flows"), base);
assert.deepEqual(parse("/flows?flow=derived-apps"), base);
assert.deepEqual(parse("/#workspace/graph"), base);
+19 -7
View File
@@ -21,6 +21,7 @@ import { MapView } from "./components/views/semantic-map";
import { RightPanel } from "./components/panels/right";
import { SearchCommand } from "./components/search-command";
import { FIND_KEY, ScreensView } from "./components/views/screens";
import { FlowsView } from "./components/views/flows";
import { publicReviewProjection, publicReviewToWorkspace } from "./public/public-workspace";
import type { PublicReview } from "../../src/contracts/capture";
import {
@@ -81,7 +82,7 @@ import {
presentScreensByPath,
screenAliasesForPresentation,
} from "./utils/screen-presentation";
import { canvasViewTransitionDirection } from "./utils/canvas-view-transition";
import { CANVAS_VIEW_ORDER, canvasViewTransitionDirection } from "./utils/canvas-view-transition";
import {
STATIC_VIEWER_SOURCE,
withStaticSourceParameters,
@@ -304,6 +305,11 @@ const ViewerApp = memo(function ViewerApp({ reportKey }: { reportKey: number })
const [workspace, setWorkspace] = useState<ViewerWorkspace>();
const [loadedPublicReview, setLoadedPublicReview] = useState<PublicReview>();
const [error, setError] = useState<string>();
// A report compare:flows wrote carries its flow review; the Flows view exists only then.
const flowReview = REPORT_SOURCE.data?.exploration?.flowReview;
const flowReviewRef = useRef(flowReview);
flowReviewRef.current = flowReview;
const views: readonly CanvasView[] = flowReview ? CANVAS_VIEW_ORDER : CANVAS_VIEW_ORDER.filter((view) => view !== "flows");
// The URL is the one record of where the viewer is; `route` is it parsed, and everything a
// reader would call navigation — canvas, rail, selection, inspector tabs, baseline, focus —
// is read off it rather than held on its own.
@@ -402,7 +408,7 @@ const ViewerApp = memo(function ViewerApp({ reportKey }: { reportKey: number })
setLoadedPublicReview(undefined);
setWorkspace(undefined);
setPayload({ map, coverage });
const nextContext = routeContextForMap(map);
const nextContext: RouteContext = { ...routeContextForMap(map), ...(REPORT_SOURCE.data.exploration.flowReview ? { flows: true } : {}) };
const nextRoute = viewerRoute(parseLocation(viewerLocation(), nextContext));
contextRef.current = nextContext;
routeRef.current = nextRoute;
@@ -969,6 +975,7 @@ const ViewerApp = memo(function ViewerApp({ reportKey }: { reportKey: number })
}, [applyCanvasPanelDefaults, navigate, requestCanvasFit]);
const selectCanvas = useCallback((canvas: CanvasView) => {
if (canvas === "flows" && !flowReviewRef.current) return;
const direction = canvasViewTransitionDirection(routeRef.current.canvas, canvas);
if (!direction) return;
if (typeof document.startViewTransition !== "function") {
@@ -1353,6 +1360,7 @@ const ViewerApp = memo(function ViewerApp({ reportKey }: { reportKey: number })
const routeContext = useMemo<RouteContext>(
() => (map ? {
...buildRouteContext,
...(flowReview ? { flows: true } : {}),
screenRoutes: screenRoutesOf(map.screens),
screenVariants: screenVariantsOf(map.screens),
...(screenAliases ? {
@@ -1361,7 +1369,7 @@ const ViewerApp = memo(function ViewerApp({ reportKey }: { reportKey: number })
screenRouteAliases: screenAliases.screenRouteAliases,
} : {}),
} : { ...buildRouteContext, screenRoutes: new Map() }),
[buildRouteContext, map, screenAliases],
[buildRouteContext, map, screenAliases, flowReview],
);
contextRef.current = routeContext;
const cameraDocumentId = currentBuild?.id ?? map?.run.id ?? payload?.map.run.id ?? "loading";
@@ -1617,11 +1625,12 @@ const ViewerApp = memo(function ViewerApp({ reportKey }: { reportKey: number })
changed: compareOrder,
selectedScreenId,
find: !phone,
flows: Boolean(flowReview),
handoff: Boolean(handoffInput),
pullRequest: review?.pullRequest,
theme: themePreference,
appearances: appearances.length > 1 ? appearances.map((option) => ({ id: option, label: appearanceLabel(option), current: option === appearance })) : [],
inspector: phone || comparingScreens ? undefined : { open: !rightCollapsed },
inspector: phone || comparingScreens || route.canvas === "flows" ? undefined : { open: !rightCollapsed },
fullscreen: !phone && fullscreenSupported() ? fullscreen : undefined,
targets: (() => {
const run = runTargets(map.run);
@@ -1696,6 +1705,7 @@ const ViewerApp = memo(function ViewerApp({ reportKey }: { reportKey: number })
onCloseInspector={closeInspector}
/>
) : null}
{route.canvas === "flows" && flowReview ? <FlowsView review={flowReview} phone={phone} /> : null}
{comparing && phone && diff && selectedScreenId && comparePicker ? (
<PhoneCompare
diff={diff}
@@ -1750,18 +1760,20 @@ const ViewerApp = memo(function ViewerApp({ reportKey }: { reportKey: number })
{phone ? (
<PhoneTabBar
canvas={route.canvas}
views={views}
compareCount={compareOrder.length}
onSelectCanvas={selectCanvas}
/>
) : <CanvasToolbar
canvas={route.canvas}
views={views}
searchOpen={searchOpen}
{...(handoffAction ? { actions: handoffAction } : {})}
more={<MoreMenu
themePreference={themePreference}
onChangeTheme={setThemePreference}
{...(fullscreenSupported() ? { fullscreen: { active: fullscreen, onToggle: toggleFullscreen } } : {})}
{...(comparingScreens ? {} : { inspector: { open: !rightCollapsed, onToggle: toggleRightPanel } })}
{...(comparingScreens || route.canvas === "flows" ? {} : { inspector: { open: !rightCollapsed, onToggle: toggleRightPanel } })}
onOpenShortcuts={() => setShortcutsOpen(true)}
/>}
onSelectCanvas={selectCanvas}
@@ -1770,7 +1782,7 @@ const ViewerApp = memo(function ViewerApp({ reportKey }: { reportKey: number })
</CanvasVisibleFrameProvider>
{shortcutsOpen ? (
<ShortcutsDialog
context={{ canvas: route.canvas, review: reviewingInSidebar, hasChanges: compareOrder.length > 0, handoff: Boolean(handoffInput), appearances: appearances.length > 1,
context={{ canvas: route.canvas, review: reviewingInSidebar, hasChanges: compareOrder.length > 0, flows: Boolean(flowReview), handoff: Boolean(handoffInput), appearances: appearances.length > 1,
fullscreen: !phone && fullscreenSupported(),
...(switchTargets ? { targets: switchTargets } : {}) }}
onClose={() => setShortcutsOpen(false)}
@@ -1787,7 +1799,7 @@ const ViewerApp = memo(function ViewerApp({ reportKey }: { reportKey: number })
onClose={closeSearch}
onSelectScreen={selectSearchScreen}
/>
{comparingScreens ? null : <RightPanel
{comparingScreens || route.canvas === "flows" ? null : <RightPanel
index={index}
screen={selectedScreen}
diff={diff}
+5 -3
View File
@@ -1,4 +1,4 @@
import { GitCompareArrows, LayoutGrid, Map, Search } from "lucide-react";
import { GitCompareArrows, LayoutGrid, Map, Search, Waypoints } from "lucide-react";
import { type CSSProperties, useLayoutEffect, useState } from "react";
import { Tooltip } from "../shared/tooltip";
import { useCanvasVisibleFrame } from "./visible-frame-provider";
@@ -10,18 +10,20 @@ import {
CANVAS_LABELS,
type CanvasView,
} from "../../utils/navigation";
import { CANVAS_VIEW_ORDER } from "../../utils/canvas-view-transition";
const VIEW_ICONS: Record<CanvasView, React.ReactNode> = {
map: <Map />,
screens: <LayoutGrid />,
compare: <GitCompareArrows />,
flows: <Waypoints />,
};
const TOOLBAR_MINIMUM_FRAME_WIDTH = 220;
type CanvasToolbarProps = {
canvas: CanvasView;
/** The views this report has, in the toolbar's order; Flows only with a flow review. */
views: readonly CanvasView[];
searchOpen: boolean;
/** Page-wide actions beside Search (Hand off to AI). */
actions?: React.ReactNode;
@@ -76,7 +78,7 @@ export function CanvasToolbar(props: CanvasToolbarProps) {
"--canvas-visible-bottom-inset": `calc(100% - ${visibleFrame.y + visibleFrame.height}px)`,
"--canvas-toolbar-shift": `${shift}px`,
} as CSSProperties) : undefined;
const views = CANVAS_VIEW_ORDER;
const views = props.views;
return (
<div ref={setElement} className="canvas-toolbar" role="toolbar" aria-label="Canvas tools" style={style}>
@@ -1,8 +1,8 @@
import { useEffect } from "react";
import { createPortal } from "react-dom";
import { X } from "lucide-react";
import { CANVAS_NUMBER_KEYS, formatShortcut, SHORTCUTS, TARGET_NUMBER_KEYS, type Shortcut } from "../../utils/shortcuts";
import type { CanvasView } from "../../utils/navigation";
import { CANVAS_LABELS, type CanvasView } from "../../utils/navigation";
import { CANVAS_NUMBER_KEYS, FLOWS_KEYS, formatShortcut, SHORTCUTS, TARGET_NUMBER_KEYS, type Shortcut } from "../../utils/shortcuts";
import { useModalDialog } from "../../hooks/use-modal-dialog";
type Row = { label: string; keys: (Shortcut | string)[] };
@@ -15,6 +15,8 @@ export type ShortcutContext = {
review: boolean;
/** The report has changed screens, so Compare and the changes menu exist. */
hasChanges: boolean;
/** The report compares two runs of a flows file, so it has a Flows view. */
flows?: boolean;
/** The report can be handed off to an agent (a hosted preview). */
handoff?: boolean;
/** The report was captured in both light and dark, so the app's appearance can be switched. */
@@ -72,12 +74,12 @@ const CAMERA: Group = {
/** The groups for one context, in two columns: what this section does, then the app-wide keys. */
export function shortcutGroups(context: ShortcutContext): Group[][] {
const sections: CanvasView[] = context.hasChanges ? ["map", "screens", "compare"] : ["map", "screens"];
const sections: CanvasView[] = ["map", "screens", ...(context.hasChanges ? ["compare" as const] : []), ...(context.flows ? ["flows" as const] : [])];
const goTo: Group = {
title: "Go to",
rows: [
{
label: context.hasChanges ? "Map, Screens, Compare" : "Map, Screens",
label: sections.map((section) => CANVAS_LABELS[section]).join(", "),
keys: sections.map((section) => SHORTCUTS.canvas[section]),
},
...(CANVAS_NUMBER_KEYS
@@ -116,8 +118,17 @@ export function shortcutGroups(context: ShortcutContext): Group[][] {
{ label: "Close the inspector, then leave the filmstrip or deselect", keys: ["Esc"] },
],
};
const flowsGroup: Group = {
title: "Flows",
rows: [
{ label: "Previous or next checkpoint", keys: [FLOWS_KEYS.previous, FLOWS_KEYS.next] },
{ label: "Next or previous changed area", keys: [FLOWS_KEYS.nextArea, FLOWS_KEYS.previousArea] },
{ label: "Flip between Before and After", keys: [FLOWS_KEYS.flip] },
],
};
const here: Group[] = context.canvas === "compare"
? [reviewGroup(context)]
: context.canvas === "flows" ? [flowsGroup]
: [...(context.review ? [reviewGroup(context)] : []), ...(context.canvas === "map" ? [mapGroup] : []),
...(context.canvas === "screens" ? [screensGroup] : []), CAMERA];
return [here, [goTo, view]];
+5 -3
View File
@@ -1,14 +1,14 @@
import { useEffect, useRef, type PointerEvent as ReactPointerEvent, type ReactNode } from "react";
import { createPortal } from "react-dom";
import { GitCompareArrows, LayoutGrid, Map as MapIcon, X } from "lucide-react";
import { GitCompareArrows, LayoutGrid, Map as MapIcon, Waypoints, X } from "lucide-react";
import { CANVAS_LABELS, type CanvasView } from "../../utils/navigation";
import { CANVAS_VIEW_ORDER } from "../../utils/canvas-view-transition";
import "./phone.css";
const VIEW_ICONS: Record<CanvasView, ReactNode> = {
map: <MapIcon />,
screens: <LayoutGrid />,
compare: <GitCompareArrows />,
flows: <Waypoints />,
};
/**
@@ -17,12 +17,14 @@ const VIEW_ICONS: Record<CanvasView, ReactNode> = {
*/
export function PhoneTabBar(props: {
canvas: CanvasView;
/** The views this report has, in the toolbar's order; Flows only with a flow review. */
views: readonly CanvasView[];
compareCount: number;
onSelectCanvas: (canvas: CanvasView) => void;
}) {
return (
<nav className="phone-tab-bar" aria-label="Views">
{CANVAS_VIEW_ORDER.map((id) => {
{props.views.map((id) => {
return (
<button key={id} type="button" className="phone-tab"
aria-current={props.canvas === id ? "page" : undefined}
+285
View File
@@ -0,0 +1,285 @@
import { useEffect, useLayoutEffect, useRef, useState, type CSSProperties, type PointerEvent, type ReactNode, type WheelEvent } from "react";
/**
* A neighbour's size and opacity by its distance from the frame in the middle. The last, invisible, place is where a
* frame waits to come in, so a swipe or a step fades it in rather than popping it.
*/
const SCALES = [1, 0.74, 0.54, 0.4, 0.3];
const OPACITIES = [1, 0.9, 0.6, 0.3, 0];
const GAP = 32;
/** A swipe has ended once its wheel events (and the trackpad's momentum after it) stop for this long. */
const SWIPE_END_MS = 100;
/**
* At the end of a swipe the strip settles on the next frame in its direction once it has moved this far towards it,
* so one wheel notch steps a frame and a twitch springs back.
*/
const SWIPE_COMMIT = 0.2;
/** How far a finger moves before a touch is a swipe rather than a tap. */
const TAP_SLOP_PX = 8;
/** How far ahead a flick's speed carries the strip when the finger lifts: its velocity times this. */
const FLICK_MS = 160;
/**
* How long the strip takes to settle after a gesture: longer the further it has left to go, so that a short way home
* is not crossed at the speed of a whole step and a long one does not snap.
*/
const settleMs = (distance: number) => Math.round(260 + 260 * Math.min(1, distance));
export type FilmstripFrame = {
/** Its place in the middle, nearer or further: content that loads lazily waits until it is near. */
middle: boolean;
distance: number;
};
export type FilmstripProps = {
/** The frames, in order: each with its width over its height. */
items: readonly { id: string; aspect: number }[];
/** The frame in the middle. */
focusedId: string;
/** Above the stage: what the middle frame is. */
header: ReactNode;
ariaLabel: string;
/** The element id a frame takes, which a page's arrow keys focus by. */
frameId?: (id: string) => string;
/** The data attribute that carries a frame's id: Screens' frames are found by `data-screen-id`. */
idAttribute?: `data-${string}`;
/** Classes a frame takes beside its own, such as its status. */
frameClass?: (id: string) => string | undefined;
/** A frame's accessible name. */
label: (id: string) => string;
/** What a frame shows. */
content: (id: string, frame: FilmstripFrame) => ReactNode;
/** Whether the frame is the one selected, for its button's pressed state. */
selected?: (id: string) => boolean;
/** Brings a frame to the middle; `replace` asks a caller that keeps history to keep it out. */
onPick: (id: string, options?: { replace?: boolean }) => void;
/** A click on the middle frame; without it the middle frame does nothing on a click. */
onOpen?: (id: string) => void;
/** The phone layout: the middle frame takes most of the width, and a finger swipes the strip. */
phone?: boolean | undefined;
};
/** The stage's size, which the frames are fitted to. */
function useSize(): [(element: HTMLDivElement | null) => void, { width: number; height: number }] {
const [element, setElement] = useState<HTMLDivElement | null>(null);
const [size, setSize] = useState({ width: 0, height: 0 });
useLayoutEffect(() => {
if (!element) return;
const measure = () => setSize({ width: element.clientWidth, height: element.clientHeight });
measure();
if (typeof ResizeObserver === "undefined") return;
const observer = new ResizeObserver(measure);
observer.observe(element);
return () => observer.disconnect();
}, [element]);
return [setElement, size];
}
/**
* Frames one at a time: the focused one large in the middle, the ones before and after it smaller and fainter the
* further they are, in order. A swipe (trackpad, wheel or finger) moves the strip with it and settles on a frame; a
* click on a neighbour brings it to the middle.
*/
export function Filmstrip(props: FilmstripProps) {
const [stageRef, stage] = useSize();
const order = props.items.map(item => item.id);
const aspects = new Map(props.items.map(item => [item.id, item.aspect]));
const at = order.indexOf(props.focusedId);
// While a swipe lasts, the strip sits at a fractional position in `order`, following the gesture.
const [swipe, setSwipe] = useState<{ position: number; direction: number } | null>(null);
// Once the gesture lets go, the strip eases the rest of the way on a timing of its own (settleMs).
const [settling, setSettling] = useState<number | null>(null);
const settled = useRef<ReturnType<typeof setTimeout>>(undefined);
// The swipe's position as of the last wheel event, which can arrive before the render that shows the one before it.
const live = useRef<number | null>(null);
// A finger on the strip: where it went down, the strip's position then, and its latest speed in px/ms.
const touch = useRef<{ id: number; x: number; from: number; lastX: number; lastAt: number; velocity: number; moved: boolean } | null>(null);
// A swipe ends with the finger lifting over a frame; the click that follows is not a tap on it.
const swallowClick = useRef(false);
const swipeEnd = useRef<ReturnType<typeof setTimeout>>(undefined);
useEffect(() => () => { clearTimeout(swipeEnd.current); clearTimeout(settled.current); }, []);
// The gesture under way: the frames it has put in the middle, the last of them, and whether it has added its one
// history entry. A frame put in the middle by anything else (an arrow key, a list, a click) ends it.
const gesture = useRef<{ shown: string; ids: Set<string>; pushed: boolean; start: number } | null>(null);
const focusedNow = useRef(props.focusedId);
useEffect(() => { focusedNow.current = props.focusedId; });
if (at < 0) return null;
// Every frame is laid out as the middle one would be, filling the stage's height short of growing wider than half
// the stage (on a phone, most of it, with the neighbours peeking in at the sides), so a landscape frame beside
// portrait ones is as wide as it may be rather than as tall; then it is scaled down around its centre, and the centres
// are spaced so the scaled frames sit a gap apart, the gap shrinking with them.
const frameSize = (id: string) => {
const aspect = aspects.get(id) ?? 1;
const height = Math.max(0, Math.min(stage.height - 16, (stage.width * (props.phone ? 0.7 : 0.5)) / aspect));
return { width: height * aspect, height };
};
// Where each frame sits with the frame at `centre` in the middle: its distance and its centre's offset.
const layout = (centre: number) => {
const placed = new Map<string, { distance: number; x: number }>();
const middle = order[centre];
if (middle === undefined) return placed;
placed.set(middle, { distance: 0, x: 0 });
for (const direction of [-1, 1]) {
let edge = frameSize(middle).width / 2;
for (let distance = 1; distance < SCALES.length; distance++) {
const id = order[centre + direction * distance];
if (id === undefined) break;
const scaled = frameSize(id).width * SCALES[distance]!;
const x = edge + GAP * SCALES[distance]! + scaled / 2;
placed.set(id, { distance, x: direction * x });
edge = x + scaled / 2;
}
}
return placed;
};
// A swipe someone else's step overtook (an arrow key, a click) no longer places the strip.
const swiping = swipe !== null && Math.round(swipe.position) === at;
const position = swiping ? swipe.position : at;
const base = Math.floor(position), fraction = position - base;
const from = layout(base), to = fraction > 0 ? layout(base + 1) : from;
const lerp = (a: number, b: number) => a + (b - a) * fraction;
const last = SCALES.length - 1;
// In the strip's order, so the frames' elements keep their order in the page as the middle changes: React moves an
// element that changes place among its siblings, and a moved element jumps instead of easing to where it goes.
const placeOf = new Map(order.map((id, place) => [id, place]));
const frames = [...new Set([...from.keys(), ...to.keys()])].sort((a, b) => placeOf.get(a)! - placeOf.get(b)!).map(id => {
const a = from.get(id), b = to.get(id);
// A frame in only one of the two places is coming in or going out at the invisible end.
const distance = lerp(a?.distance ?? last, b?.distance ?? last);
const x = lerp(a?.x ?? b!.x, b?.x ?? a!.x);
const scale = lerp(SCALES[a?.distance ?? last]!, SCALES[b?.distance ?? last]!);
const opacity = lerp(OPACITIES[a?.distance ?? last]!, OPACITIES[b?.distance ?? last]!);
return { id, distance, x, scale, opacity, ...frameSize(id) };
});
// The frames' gap at rest, which one frame of swipe travels.
const pitch = Math.abs(layout(at).get(order[at + 1] ?? order[at - 1] ?? "")?.x ?? frameSize(props.focusedId).width) || 1;
const clamp = (value: number) => Math.max(0, Math.min(order.length - 1, value));
const abandon = () => { clearTimeout(swipeEnd.current); gesture.current = null; live.current = null; };
const overtaken = () => !gesture.current || !gesture.current.ids.has(focusedNow.current);
// Where a gesture starts: where the one under way has the strip, or the frame in the middle for a new one.
const begin = (): number => {
if (!overtaken() && live.current !== null) return live.current;
abandon();
const index = order.indexOf(focusedNow.current);
const start = index < 0 ? at : index;
gesture.current = { shown: focusedNow.current, ids: new Set([focusedNow.current]), pushed: false, start };
return start;
};
// Puts a frame in the middle for the gesture: its first pick is one step back in the history, the rest replace it.
const show = (id: string) => {
const current = gesture.current;
if (!current || id === current.shown) return;
current.shown = id; current.ids.add(id);
props.onPick(id, current.pushed ? { replace: true } : undefined);
current.pushed = true;
};
// Puts the strip at `next` while a gesture lasts; the middle frame, and with it the header, follows the swipe as it
// passes halfway.
const moveTo = (next: number, direction: number) => {
live.current = next;
setSwipe({ position: next, direction });
show(order[Math.round(next)]!);
};
// When the gesture ends, the strip settles on the next frame in its direction once it is SWIPE_COMMIT of the way
// there; `next` may be ahead of where the strip is, carried on by a flick.
const settle = (next: number, direction: number) => {
if (overtaken()) { abandon(); setSwipe(null); return; }
const index = clamp(direction > 0 ? Math.floor(next + 1 - SWIPE_COMMIT) : direction < 0 ? Math.ceil(next - 1 + SWIPE_COMMIT) : Math.round(next));
show(order[index]!);
gesture.current = null;
const ms = settleMs(Math.abs(index - (live.current ?? next)));
clearTimeout(settled.current);
setSettling(ms);
settled.current = setTimeout(() => setSettling(null), ms);
live.current = null;
setSwipe(null);
};
// A swipe along either axis moves the strip with it, so a mouse wheel works as well as a trackpad; it has ended when
// its events stop.
const onWheel = (event: WheelEvent) => {
// A pinch arrives as a wheel with Control held; it zooms the page, not the strip.
if (event.ctrlKey) return;
const delta = Math.abs(event.deltaX) > Math.abs(event.deltaY) ? event.deltaX : event.deltaY;
const pixels = event.deltaMode === 1 ? delta * 16 : event.deltaMode === 2 ? delta * stage.width : delta;
if (!pixels) return;
const next = clamp(begin() + pixels / pitch);
// The way the whole swipe went, not its last event: momentum's last 1px back must not turn a swipe around.
const direction = Math.sign(next - gesture.current!.start) || Math.sign(pixels);
moveTo(next, direction);
clearTimeout(swipeEnd.current);
swipeEnd.current = setTimeout(() => settle(next, direction), SWIPE_END_MS);
};
// A finger drags the strip under it; on lifting, a flick carries it on by its speed before it settles. A mouse
// keeps its clicks and wheel.
const onPointerDown = (event: PointerEvent) => {
// A browser that sent no click after the last swipe must not have this press's click swallowed, by finger or mouse.
swallowClick.current = false;
// A touch whose end never reached the strip (it slid off before it moved enough to be held) gives way to this one.
if (event.pointerType === "mouse" || touch.current?.id === event.pointerId) return;
touch.current = { id: event.pointerId, x: event.clientX, from: begin(), lastX: event.clientX, lastAt: event.timeStamp, velocity: 0, moved: false };
};
const onPointerMove = (event: PointerEvent) => {
const current = touch.current;
if (!current || current.id !== event.pointerId) return;
const dx = event.clientX - current.x;
if (!current.moved && Math.abs(dx) < TAP_SLOP_PX) return;
// A wheel swipe's settle waits until the finger takes over; a tap leaves it to settle.
if (!current.moved) { current.moved = true; clearTimeout(swipeEnd.current); event.currentTarget.setPointerCapture(event.pointerId); }
const elapsed = event.timeStamp - current.lastAt;
if (elapsed > 0) current.velocity = (event.clientX - current.lastX) / elapsed;
current.lastX = event.clientX; current.lastAt = event.timeStamp;
moveTo(clamp(current.from - dx / pitch), -Math.sign(dx));
};
const onPointerEnd = (event: PointerEvent) => {
const current = touch.current;
if (!current || current.id !== event.pointerId) return;
touch.current = null;
if (!current.moved) return;
swallowClick.current = true;
// A finger that stopped before lifting carries nothing on.
const velocity = event.timeStamp - current.lastAt > 80 ? 0 : current.velocity;
const next = clamp((live.current ?? position) - (velocity * FLICK_MS) / pitch);
settle(next, velocity ? -Math.sign(velocity) : -Math.sign(event.clientX - current.x));
};
return (
<section className={`screens-strip${props.phone ? " phone" : ""}${props.onOpen ? "" : " static-middle"}`} aria-label={props.ariaLabel} onWheel={onWheel}
onPointerDown={onPointerDown} onPointerMove={onPointerMove} onPointerUp={onPointerEnd} onPointerCancel={onPointerEnd}
onClickCapture={(event) => { if (swallowClick.current) { swallowClick.current = false; event.preventDefault(); event.stopPropagation(); } }}>
{props.header}
<div ref={stageRef} className={`screens-strip-stage${swiping ? " swiping" : settling ? " settling" : ""}`}
style={settling ? ({ "--strip-settle-ms": `${settling}ms` } as CSSProperties) : undefined}>
{stage.width > 0 ? frames.map(({ id, distance, x, scale, opacity, width, height }) => {
const middle = id === props.focusedId;
const hidden = opacity === 0;
const extra = props.frameClass?.(id);
const label = props.label(id);
return (
<div key={id} {...(props.frameId ? { id: props.frameId(id) } : {})} {...{ [props.idAttribute ?? "data-frame-id"]: id }}
className={`screens-strip-frame${middle ? " focused" : ""}${extra ? ` ${extra}` : ""}`}
style={{
width,
height,
opacity,
zIndex: Math.round((SCALES.length - distance) * 10),
transform: `translate(-50%, -50%) translateX(${x}px) scale(${scale})`,
}}>
{/* Only the middle frame is in the tab order: the arrows step the strip, and focus on a neighbour the stage
clips would scroll it off its centre. */}
<button type="button" className="screens-strip-target" aria-label={label} title={middle ? undefined : label}
tabIndex={middle ? undefined : -1} aria-hidden={hidden || undefined} disabled={hidden}
aria-pressed={props.selected ? props.selected(id) : undefined} aria-current={middle ? "true" : undefined}
onClick={() => (middle ? props.onOpen?.(id) : props.onPick(id))}>
{props.content(id, { middle, distance })}
</button>
</div>
);
}) : null}
</div>
</section>
);
}
+113
View File
@@ -0,0 +1,113 @@
/* The Flows view: the Screens sidebar on the left, one flow on the stage, the flow bar where Compare keeps its bar. */
.flows-page {
--flows-right-inset: 0px;
--flows-mono: "JetBrains Mono", ui-monospace, SFMono-Regular, Menlo, monospace;
position: absolute;
inset: 0;
overflow: hidden;
padding: calc(var(--chrome-top) + var(--chrome-height) + 20px) calc(var(--flows-right-inset) + 28px) 96px var(--canvas-chrome-inline-inset);
color: var(--text);
background: var(--screen);
}
.flows-body { display: flex; height: 100%; align-items: flex-start; gap: 32px; }
.flows-side { max-height: 100%; }
.flows-find { margin: 0 0 8px; }
.flows-mark { flex: none; }
/* ── The list: each flow with its status line; the open flow's checkpoints under it ── */
.flows-list, .flows-checkpoints { margin: 0; padding: 0; list-style: none; }
.flows-list { display: flex; flex-direction: column; gap: 1px; }
.flows-row {
display: flex;
width: 100%;
align-items: flex-start;
gap: 8px;
padding: 6px 8px;
border-radius: 6px;
color: var(--text);
background: transparent;
text-align: left;
}
.flows-row > .flows-mark { margin-top: 2px; }
.flows-row:hover { background: var(--panel-hover); }
.flows-row-text { display: flex; min-width: 0; flex: 1; flex-direction: column; gap: 1px; }
.flows-row-name { overflow: hidden; font-size: 13px; text-overflow: ellipsis; white-space: nowrap; }
.flows-row.open .flows-row-name { font-weight: 600; }
.flows-row-status { color: var(--secondary); font-size: 12px; }
.flows-row-status.stopped { color: var(--warning); }
.flows-checkpoints { padding: 2px 0 6px; }
.flows-checkpoint { padding-left: 30px; color: var(--text); }
/* ── The flow bar: the flow's name and Before | After ── */
.flows-top { z-index: 6; }
.flows-bar { width: auto; gap: 10px; padding-left: 14px; }
.flows-bar-name { flex: none; font-size: 14px; font-weight: 600; white-space: nowrap; }
.flows-bar .toggle-group { width: auto; flex: none; }
.flows-bar .toggle-row { width: auto; }
.flows-bar .toggle-row .tooltip-anchor { flex: none; }
.flows-bar .toggle-button { padding: 0 12px; white-space: nowrap; }
.flows-bar .toggle-label { overflow-wrap: normal; }
/* ── The strip: Screens' filmstrip, a checkpoint to a frame ── */
/* A sideways swipe steps the strip; it must not also take the browser back a page. */
html:has(.flows-page) { overscroll-behavior-x: none; }
.flows-body > .screens-strip { height: 100%; }
/* The middle checkpoint opens nothing: it is already as large as the page shows it. */
.screens-strip.static-middle .screens-strip-frame.focused .screens-strip-target { cursor: default; }
/* As on Screens, the middle frame keeps its screenshot's own colours when the arrows focus it. */
.flows-page .screens-strip-target:focus-visible { filter: none; }
.flows-strip-head { display: flex; width: 100%; flex: none; flex-direction: column; align-items: center; gap: 2px; }
.flows-step.exact .screens-strip-crumb-label { font-family: var(--flows-mono); font-size: 12px; }
.flows-step.failed { color: var(--text); font-weight: 500; }
.flows-step.not-run { opacity: 0.6; }
.flows-word { flex: none; color: var(--secondary); font-size: 13px; font-weight: 600; }
.flows-word.changed { color: var(--modified); }
.flows-word.added { color: var(--success); }
.flows-word.removed { color: var(--danger); }
.flows-word.not-comparable, .flows-word.stopped, .flows-word.not-reached { color: var(--warning); }
.flows-strip-note {
max-width: 100%;
margin: 0;
overflow: hidden;
color: var(--secondary);
font-size: 13px;
text-overflow: ellipsis;
white-space: nowrap;
}
.flows-strip-note.stopped { color: var(--warning); }
.flows-shot { position: relative; display: block; width: 100%; height: 100%; }
.flows-shot img { display: block; width: 100%; height: 100%; object-fit: contain; }
.flows-shot.missing {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
gap: 8px;
padding: 16px;
color: var(--secondary);
font-size: 13px;
font-weight: 500;
text-align: center;
}
.flows-area {
position: absolute;
margin: -3px;
padding: 3px;
border-radius: 4px;
outline: 2px solid var(--modified);
background: color-mix(in srgb, var(--modified) 9%, transparent);
box-sizing: content-box;
pointer-events: none;
}
.flows-area.current { outline-width: 3px; background: color-mix(in srgb, var(--modified) 18%, transparent); }
.flows-page.phone { padding: calc(var(--chrome-top) + var(--chrome-height) + 12px) 12px 96px; overflow-y: auto; }
/* On a phone the bar is the page's first row, under the pill, with the flow's name and Before | After. */
.flows-page.phone .flows-top { position: static; max-width: none; margin: 0 0 12px; transform: none; }
.flows-page.phone .flows-bar { width: 100%; zoom: 1; }
.flows-page.phone .flows-bar-name { min-width: 0; flex: 1; overflow: hidden; text-overflow: ellipsis; }
.flows-page.phone .flows-body { flex-direction: column; height: auto; }
.flows-page.phone .flows-side { width: 100%; }
/* On a phone the strip comes first, as tall as most of the screen, and the list of flows follows it. */
.flows-page.phone .flows-body > .screens-strip { order: -1; width: 100%; height: 72svh; flex: none; }
+266
View File
@@ -0,0 +1,266 @@
import { useEffect, useMemo, useRef, useState, type ReactNode } from "react";
import { ChevronRight } from "lucide-react";
import type { FlowReview, FlowReviewCheckpoint, FlowReviewFlow } from "../../../../src/contracts/flow-comparison";
import { embeddedArtifactUrl } from "../../utils/embedded-report";
import {
checkpointLabel, checkpointMark, firstProblem, flowMark, flowNeedsReview, flowStatusLine, sidePicture, stepsBefore, stoppedEarly,
textChangeLine, visibleFlows, type FlowFilter, type FlowMark, type FlowSide,
} from "../../utils/flows-model";
import { normalizeSearchText } from "../../utils/search";
import { FLOWS_KEYS, formatShortcut, matchesShortcut, shortcutsLive } from "../../utils/shortcuts";
import { useCanvasVisibleFrame } from "../canvas/visible-frame-provider";
import { Filmstrip } from "../shared/filmstrip";
import { PanelSearch } from "../shared/panel-search";
import { DEFAULT_SCREENSHOT_RATIO, ScreenshotImage } from "../shared/screenshot-image";
import { ToggleGroup } from "../shared/toggle-group";
import "./screens.css";
import "./flows.css";
/** A neighbour's outline in the strip, in its status colour, as Screens' filmstrip outlines a changed neighbour. */
const FRAME_CLASS: Partial<Record<FlowMark, string>> = { changed: "diff-modified", added: "diff-added", removed: "diff-removed" };
/** A status by the shape of its icon as well as its colour, so no state depends on colour alone. */
export function FlowMarkIcon(props: { mark: FlowMark; size?: number }) {
const size = props.size ?? 14, c = size / 2, r = size / 2 - 1.5;
const body: Record<FlowMark, ReactNode> = {
changed: <><circle cx={c} cy={c} r={r} fill="none" stroke="var(--modified)" strokeWidth="1.5" /><circle cx={c} cy={c} r={r - 2.6} fill="var(--modified)" /></>,
unchanged: <><circle cx={c} cy={c} r={r} fill="none" stroke="var(--border)" strokeWidth="1.5" /><path d={`M${c - 2.4} ${c}l1.7 1.7 3.1-3.3`} fill="none" stroke="var(--tertiary)" strokeWidth="1.4" strokeLinecap="round" strokeLinejoin="round" /></>,
"not-reached": <circle cx={c} cy={c} r={r} fill="none" stroke="var(--tertiary)" strokeWidth="1.5" strokeDasharray="2 2.2" />,
stopped: <><circle cx={c} cy={c} r={r + 0.5} fill="var(--warning-soft)" stroke="var(--warning)" strokeWidth="1.2" /><path d={`M${c - 2.1} ${c - 2.1}l4.2 4.2M${c + 2.1} ${c - 2.1}l-4.2 4.2`} stroke="var(--warning)" strokeWidth="1.5" strokeLinecap="round" /></>,
added: <><circle cx={c} cy={c} r={r + 0.75} fill="var(--success)" /><path d={`M${c} ${c - 2.6}v5.2M${c - 2.6} ${c}h5.2`} stroke="var(--screen)" strokeWidth="1.5" strokeLinecap="round" /></>,
removed: <><circle cx={c} cy={c} r={r + 0.75} fill="var(--danger)" /><path d={`M${c - 2.6} ${c}h5.2`} stroke="var(--screen)" strokeWidth="1.5" strokeLinecap="round" /></>,
"not-comparable": <><path d={`M${c} 1.5 ${size - 1.5} ${size - 2} H1.5Z`} fill="var(--warning-soft)" stroke="var(--warning)" strokeWidth="1.2" strokeLinejoin="round" /><path d={`M${c} ${c - 1.5}v2.6`} stroke="var(--warning)" strokeWidth="1.4" strokeLinecap="round" /></>,
};
return <svg className="flows-mark" width={size} height={size} viewBox={`0 0 ${size} ${size}`} aria-hidden="true">{body[props.mark]}</svg>;
}
/** One checkpoint's picture on one side, with its changed areas outlined, or why it has none. */
function CheckpointPicture(props: { checkpoint: FlowReviewCheckpoint; side: FlowSide; areas: boolean; area?: number | undefined; eager: boolean }) {
const picture = sidePicture(props.side === "after" ? props.checkpoint.after : props.checkpoint.before);
if (!picture || "missing" in picture) {
const why = !picture ? (props.side === "after" ? "Not in this build's flows file" : "Not in the earlier build's flows file") : "Not reached";
return (
<span className="flows-shot missing">
<FlowMarkIcon mark="not-reached" size={20} />
<span>{why}</span>
</span>
);
}
return (
<span className="flows-shot">
<ScreenshotImage src={embeddedArtifactUrl(picture.sha256)} alt="" draggable={false} decoding="async" loading={props.eager ? "eager" : "lazy"} />
{props.areas ? (props.checkpoint.regions ?? []).map((region, place) => (
<span key={place} className={`flows-area${props.area === place ? " current" : ""}`} aria-hidden="true"
style={{ left: `${region.x}%`, top: `${region.y}%`, width: `${region.width}%`, height: `${region.height}%` }} />
)) : null}
</span>
);
}
/**
* Above the strip: the steps that led to the checkpoint in the middle, then its name and status, as the Screens
* filmstrip shows the path to its screen. One line under it says what the picture cannot: the text a small change
* changed, or why the run stopped.
*/
function StripHeader(props: { flow: FlowReviewFlow; index: number }) {
const checkpoint = props.flow.checkpoints[props.index]!;
const mark = checkpointMark(checkpoint);
const steps = stepsBefore(props.flow, props.index);
const failed = steps.some(step => step.failed);
const note = failed && props.flow.after?.reason ? props.flow.after.reason : textChangeLine(checkpoint);
return (
<div className="flows-strip-head">
<div className="screens-strip-path">
{steps.map(({ step, failed: stoppedHere, ran }, place) => {
const text = "action" in step ? step.action : step.do;
return (
<span key={place} className={`screens-strip-crumb flows-step${"action" in step ? " exact" : ""}${stoppedHere ? " failed" : ""}${ran ? "" : " not-run"}`} title={text}>
{stoppedHere ? <FlowMarkIcon mark="stopped" size={13} /> : null}
<span className="screens-strip-crumb-label">{text}</span>
<ChevronRight aria-hidden="true" />
</span>
);
})}
<span className="screens-strip-crumb current" title={checkpoint.name}>
<FlowMarkIcon mark={mark} size={12} />
<span className="screens-strip-crumb-label">{checkpoint.name}</span>
{checkpoint.verdict === "unchanged" ? null : <span className={`flows-word ${mark}`}>{checkpointLabel(checkpoint)}</span>}
</span>
</div>
{note ? <p className={`flows-strip-note${failed ? " stopped" : ""}`} title={note}>{note}</p> : null}
</div>
);
}
function matches(flow: FlowReviewFlow, query: string): boolean {
if (!query) return true;
const haystack = [flow.name, ...flow.checkpoints.map((checkpoint) => checkpoint.name)];
return haystack.some((text) => normalizeSearchText(text).includes(query));
}
/**
* The Flows view: a report that compares two runs of a flows file, one flow at a time, as a filmstrip of its
* checkpoints in order. A flow opens at its first problem. Before | After (X) flips every picture to the other build in
* place. Nothing is saved: the filter, the flow and the checkpoint are this page's state.
*/
export function FlowsView(props: { review: FlowReview; phone: boolean }) {
const { review } = props;
const anyChanged = review.flows.some(flowNeedsReview);
const [filter, setFilter] = useState<FlowFilter>(anyChanged ? "changed" : "all");
const [query, setQuery] = useState("");
const shown = useMemo(() => {
const needle = normalizeSearchText(query.trim());
return visibleFlows(review, filter).filter((flow) => matches(flow, needle));
}, [review, filter, query]);
const [flowName, setFlowName] = useState(() => visibleFlows(review, anyChanged ? "changed" : "all")[0]?.name ?? review.flows[0]?.name);
const flow = review.flows.find((item) => item.name === flowName) ?? shown[0] ?? review.flows[0];
const [index, setIndex] = useState(() => (flow ? firstProblem(flow) : 0));
const [side, setSide] = useState<FlowSide>("after");
const [area, setArea] = useState<number | undefined>(undefined);
const { visibleFrame } = useCanvasVisibleFrame();
const openFlow = (next: FlowReviewFlow) => {
setFlowName(next.name);
setIndex(firstProblem(next));
setArea(undefined);
};
const openCheckpoint = (place: number) => {
setIndex(place);
setArea(undefined);
};
// The arrows step the strip, J and K walk the changed areas of the checkpoint in the middle, X flips.
const state = useRef({ flow, index, area });
state.current = { flow, index, area };
useEffect(() => {
const onKeyDown = (event: KeyboardEvent) => {
if (!shortcutsLive(event)) return;
const { flow: current, index: place, area: shownArea } = state.current;
if (!current) return;
const regions = current.checkpoints[place]?.regions?.length ?? 0;
if (matchesShortcut(event, FLOWS_KEYS.flip)) setSide((value) => (value === "after" ? "before" : "after"));
else if (matchesShortcut(event, FLOWS_KEYS.next) || matchesShortcut(event, FLOWS_KEYS.previous)) {
const next = place + (matchesShortcut(event, FLOWS_KEYS.next) ? 1 : -1);
if (next < 0 || next >= current.checkpoints.length) return;
setIndex(next); setArea(undefined);
} else if (matchesShortcut(event, FLOWS_KEYS.nextArea) || matchesShortcut(event, FLOWS_KEYS.previousArea)) {
if (!regions) return;
const forward = matchesShortcut(event, FLOWS_KEYS.nextArea);
setArea(shownArea === undefined ? (forward ? 0 : regions - 1) : (shownArea + (forward ? 1 : regions - 1)) % regions);
} else return;
event.preventDefault();
};
window.addEventListener("keydown", onKeyDown);
return () => window.removeEventListener("keydown", onKeyDown);
}, []);
if (!flow) {
return (
<div className="graph-canvas canvas-empty">
<div className="page-empty compact-empty map-empty"><h2>No flows</h2><p>The runs compared have no flows in common.</p></div>
</div>
);
}
const rightInset = visibleFrame ? `calc(100% - ${visibleFrame.x + visibleFrame.width}px)` : "0px";
const sidebar = (
<aside className="screens-side flows-side">
<nav className="screens-tree" aria-label="Flows">
{props.phone ? null : (
<div className="screens-find flows-find">
<PanelSearch value={query} placeholder="Search" ariaLabel="Find flows" clearLabel="Clear find query" name="flows-find"
onEscape={() => setQuery("")} onChange={setQuery} />
</div>
)}
{anyChanged ? (
<div className="screens-filter" role="group" aria-label="Show">
{(["changed", "all"] as const).map((value) => (
<button key={value} type="button" className="screens-chip" aria-pressed={filter === value} onClick={() => setFilter(value)}>
{value === "changed" ? "Changed" : "All"}
<span className="screens-chip-count">{visibleFlows(review, value).length}</span>
</button>
))}
</div>
) : null}
<ul className="flows-list">
{shown.map((item) => {
const open = item.name === flow.name;
return (
<li key={item.name}>
<button type="button" className={`flows-row${open ? " open" : ""}`} aria-expanded={open} onClick={() => openFlow(item)}>
<FlowMarkIcon mark={flowMark(item)} />
<span className="flows-row-text">
<span className="flows-row-name">{item.name}</span>
<span className={`flows-row-status${stoppedEarly(item) ? " stopped" : ""}`}>{flowStatusLine(item)}</span>
</span>
</button>
{open ? (
<ul className="flows-checkpoints">
{item.checkpoints.map((entry, place) => (
<li key={entry.name}>
<button type="button" className="screens-tree-row flows-checkpoint" aria-current={place === index}
onClick={() => openCheckpoint(place)}>
<FlowMarkIcon mark={checkpointMark(entry)} size={12} />
<span className="screens-tree-label">{entry.name}</span>
</button>
</li>
))}
</ul>
) : null}
</li>
);
})}
</ul>
{shown.length === 0 ? <p className="screens-find-empty" role="status">No flow matches</p> : null}
</nav>
</aside>
);
// The flow's name and the build its pictures show.
const flowBar = (
<div className="compare-top flows-top">
<div className="compare-bar flows-bar" role="toolbar" aria-label="Flow">
<FlowMarkIcon mark={flowMark(flow)} />
<strong className="flows-bar-name">{flow.name}</strong>
<ToggleGroup<FlowSide>
ariaLabel="Build"
value={side}
arrowKeys={false}
onChange={setSide}
options={[
{ id: "before", label: "Before", shortcut: formatShortcut(FLOWS_KEYS.flip) },
{ id: "after", label: "After", shortcut: formatShortcut(FLOWS_KEYS.flip) },
]}
/>
</div>
</div>
);
const checkpoints = flow.checkpoints;
const strip = (
<Filmstrip
key={flow.name}
items={checkpoints.map((_, place) => ({ id: String(place), aspect: DEFAULT_SCREENSHOT_RATIO }))}
focusedId={String(Math.min(index, checkpoints.length - 1))}
header={<StripHeader flow={flow} index={Math.min(index, checkpoints.length - 1)} />}
ariaLabel={`${flow.name}, in order`}
phone={props.phone}
frameClass={(id) => FRAME_CLASS[checkpointMark(checkpoints[Number(id)]!)]}
label={(id) => checkpoints[Number(id)]!.name}
onPick={(id) => openCheckpoint(Number(id))}
content={(id, { middle, distance }) => (
<CheckpointPicture checkpoint={checkpoints[Number(id)]!} side={side} areas={middle} {...(middle ? { area } : {})} eager={distance <= 1.5} />
)}
/>
);
return (
<main className={`flows-page${props.phone ? " phone" : ""}`} aria-label="Flows" style={{ "--flows-right-inset": rightInset } as React.CSSProperties}>
{flowBar}
<div className="flows-body">
{sidebar}
{strip}
</div>
</main>
);
}
+45 -265
View File
@@ -1,35 +1,13 @@
import { useEffect, useLayoutEffect, useMemo, useRef, useState, type CSSProperties, type PointerEvent, type WheelEvent } from "react";
import { useMemo } from "react";
import { ChevronRight } from "lucide-react";
import { artifactUrl, isLaunchScreen, type MapIndex, type Screen } from "../../utils/model";
import { primaryScreenshotForScreen, screenshotSize, type ScreenshotSize } from "../../utils/screen-images";
import { diffClass, type BuildDiff } from "../../utils/build-diff";
import { screenLabel } from "../../utils/screen-path";
import { treeNodeKey, type ScreenTree, type ScreenTreeIndex, type ScreenTreeNode } from "../../utils/screen-tree";
import { Filmstrip } from "../shared/filmstrip";
import { ScreenshotImage } from "../shared/screenshot-image";
/**
* A neighbour's size and opacity by its distance from the screen in the middle. The last, invisible, place is where a
* screen waits to come in, so a swipe or a step fades it in rather than popping it.
*/
const SCALES = [1, 0.74, 0.54, 0.4, 0.3];
const OPACITIES = [1, 0.9, 0.6, 0.3, 0];
const GAP = 32;
/** A swipe has ended once its wheel events (and the trackpad's momentum after it) stop for this long. */
const SWIPE_END_MS = 100;
/**
* At the end of a swipe the strip settles on the next screen in its direction once it has moved this far towards it,
* so one wheel notch steps a screen and a twitch springs back.
*/
const SWIPE_COMMIT = 0.2;
/** How far a finger moves before a touch is a swipe rather than a tap. */
const TAP_SLOP_PX = 8;
/** How far ahead a flick's speed carries the strip when the finger lifts: its velocity times this. */
const FLICK_MS = 160;
/**
* How long the strip takes to settle after a gesture: longer the further it has left to go, so that a short way home
* is not crossed at the speed of a whole step and a long one does not snap.
*/
const settleMs = (distance: number) => Math.round(260 + 260 * Math.min(1, distance));
/** An iPhone's portrait shape, for a run with no screenshot sizes at all. */
const FALLBACK_SIZE: ScreenshotSize = { imageWidth: 390, imageHeight: 844 };
@@ -61,50 +39,12 @@ function labelOf(screen: Screen, index: MapIndex): string {
return isLaunchScreen(screen) ? "Cold start" : screenLabel(screen, index);
}
/** The stage's size, which the frames are fitted to. */
function useSize(): [(element: HTMLDivElement | null) => void, { width: number; height: number }] {
const [element, setElement] = useState<HTMLDivElement | null>(null);
const [size, setSize] = useState({ width: 0, height: 0 });
useLayoutEffect(() => {
if (!element) return;
const measure = () => setSize({ width: element.clientWidth, height: element.clientHeight });
measure();
if (typeof ResizeObserver === "undefined") return;
const observer = new ResizeObserver(measure);
observer.observe(element);
return () => observer.disconnect();
}, [element]);
return [setElement, size];
}
/**
* Screens one at a time: the focused screen large in the middle, the screens before and after it smaller and fainter
* the further they are, in the tree's order. Above it, the rows of the tree it was opened from, ending in its name with
* its status dot. The neighbours, whose names are not shown, are outlined in their status colour instead.
* Screens one at a time, in the tree's order. Above them, the rows of the tree the middle screen was opened from,
* ending in its name with its status dot. The neighbours, whose names are not shown, are outlined in their status colour
* instead.
*/
export function ScreensFilmstrip(props: ScreensFilmstripProps) {
const [stageRef, stage] = useSize();
const at = props.order.indexOf(props.focusedId);
// While a swipe lasts, the strip sits at a fractional position in `order`, following the gesture.
const [swipe, setSwipe] = useState<{ position: number; direction: number } | null>(null);
// Once the gesture lets go, the strip eases the rest of the way on a timing of its own (settleMs).
const [settling, setSettling] = useState<number | null>(null);
const settled = useRef<ReturnType<typeof setTimeout>>(undefined);
// The swipe's position as of the last wheel event, which can arrive before the render that shows the one before it.
const live = useRef<number | null>(null);
// A finger on the strip: where it went down, the strip's position then, and its latest speed in px/ms.
const touch = useRef<{ id: number; x: number; from: number; lastX: number; lastAt: number; velocity: number; moved: boolean } | null>(null);
// A swipe ends with the finger lifting over a screen; the click that follows is not a tap on it.
const swallowClick = useRef(false);
const swipeEnd = useRef<ReturnType<typeof setTimeout>>(undefined);
useEffect(() => () => { clearTimeout(swipeEnd.current); clearTimeout(settled.current); }, []);
// The gesture under way: the screens it has put in the middle, the last of them, and whether it has added its one
// history entry. A screen put in the middle by anything else (an arrow key, the tree, a click) ends it.
const gesture = useRef<{ shown: string; ids: Set<string>; pushed: boolean; start: number } | null>(null);
const focusedNow = useRef(props.focusedId);
useEffect(() => { focusedNow.current = props.focusedId; });
const focused = props.index.screens.get(props.focusedId);
// A row of the tree by its fold key, as the path above the frame names it.
const rowLabels = useMemo(() => {
const labels = new Map<string, string>();
@@ -121,214 +61,54 @@ export function ScreensFilmstrip(props: ScreensFilmstripProps) {
walk(props.tree.tree);
return labels;
}, [props.index, props.tree]);
const focused = props.index.screens.get(props.focusedId);
if (!focused) return null;
if (!focused || at < 0) return null;
const sizeOf = (screen: Screen) => screenshotSize(primaryScreenshotForScreen(screen, props.index)) ?? props.typicalSize ?? FALLBACK_SIZE;
const aspect = (screen: Screen) => {
const size = sizeOf(screen);
return size.imageWidth / size.imageHeight;
};
// Every frame is laid out as the middle one would be, filling the stage's height short of growing wider than half
// the stage (on a phone, most of it, with the neighbours peeking in at the sides), so a landscape capture beside portrait ones is as wide as it may be rather than as tall; then it is scaled
// down around its centre, and the centres are spaced so the scaled frames sit a gap apart, the gap shrinking with them.
const frameSize = (screen: Screen) => {
const height = Math.max(0, Math.min(stage.height - 16, (stage.width * (props.phone ? 0.7 : 0.5)) / aspect(screen)));
return { width: height * aspect(screen), height };
};
// Where each screen sits with the screen at `centre` in the middle: its distance and its centre's offset.
const layout = (centre: number) => {
const placed = new Map<string, { distance: number; x: number }>();
const middle = props.index.screens.get(props.order[centre] ?? "");
if (!middle) return placed;
placed.set(props.order[centre]!, { distance: 0, x: 0 });
for (const direction of [-1, 1]) {
let edge = frameSize(middle).width / 2;
for (let distance = 1; distance < SCALES.length; distance++) {
const id = props.order[centre + direction * distance];
const screen = id ? props.index.screens.get(id) : undefined;
if (!id || !screen) break;
const scaled = frameSize(screen).width * SCALES[distance]!;
const x = edge + GAP * SCALES[distance]! + scaled / 2;
placed.set(id, { distance, x: direction * x });
edge = x + scaled / 2;
}
}
return placed;
};
// A swipe someone else's step overtook (an arrow key, a click) no longer places the strip.
const swiping = swipe !== null && Math.round(swipe.position) === at;
const position = swiping ? swipe.position : at;
const base = Math.floor(position), fraction = position - base;
const from = layout(base), to = fraction > 0 ? layout(base + 1) : from;
const lerp = (a: number, b: number) => a + (b - a) * fraction;
const last = SCALES.length - 1;
// In the strip's order, so the frames' elements keep their order in the page as the middle changes: React moves an
// element that changes place among its siblings, and a moved element jumps instead of easing to where it goes.
const placeOf = new Map(props.order.map((id, place) => [id, place]));
const frames = [...new Set([...from.keys(), ...to.keys()])].sort((a, b) => placeOf.get(a)! - placeOf.get(b)!).flatMap(id => {
const items = props.order.flatMap(id => {
const screen = props.index.screens.get(id);
if (!screen) return [];
const a = from.get(id), b = to.get(id);
// A screen in only one of the two places is coming in or going out at the invisible end.
const distance = lerp(a?.distance ?? last, b?.distance ?? last);
const x = lerp(a?.x ?? b!.x, b?.x ?? a!.x);
const scale = lerp(SCALES[a?.distance ?? last]!, SCALES[b?.distance ?? last]!);
const opacity = lerp(OPACITIES[a?.distance ?? last]!, OPACITIES[b?.distance ?? last]!);
return [{ id, screen, distance, x, scale, opacity, ...frameSize(screen) }];
const size = screenshotSize(primaryScreenshotForScreen(screen, props.index)) ?? props.typicalSize ?? FALLBACK_SIZE;
return [{ id, aspect: size.imageWidth / size.imageHeight }];
});
// The frames' gap at rest, which one screen of swipe travels.
const pitch = Math.abs(layout(at).get(props.order[at + 1] ?? props.order[at - 1] ?? "")?.x ?? frameSize(focused).width) || 1;
const clamp = (value: number) => Math.max(0, Math.min(props.order.length - 1, value));
const abandon = () => { clearTimeout(swipeEnd.current); gesture.current = null; live.current = null; };
const overtaken = () => !gesture.current || !gesture.current.ids.has(focusedNow.current);
// Where a gesture starts: where the one under way has the strip, or the screen in the middle for a new one.
const begin = (): number => {
if (!overtaken() && live.current !== null) return live.current;
abandon();
const index = props.order.indexOf(focusedNow.current);
const start = index < 0 ? at : index;
gesture.current = { shown: focusedNow.current, ids: new Set([focusedNow.current]), pushed: false, start };
return start;
};
// Puts a screen in the middle for the gesture: its first pick is one step back in the history, the rest replace it.
const show = (id: string) => {
const current = gesture.current;
if (!current || id === current.shown) return;
current.shown = id; current.ids.add(id);
props.onPick(id, current.pushed ? { replace: true } : undefined);
current.pushed = true;
};
// Puts the strip at `next` while a gesture lasts; the middle screen, and with it the path above, follows the swipe as
// it passes halfway.
const moveTo = (next: number, direction: number) => {
live.current = next;
setSwipe({ position: next, direction });
show(props.order[Math.round(next)]!);
};
// When the gesture ends, the strip settles on the next screen in its direction once it is SWIPE_COMMIT of the way
// there; `next` may be ahead of where the strip is, carried on by a flick.
const settle = (next: number, direction: number) => {
if (overtaken()) { abandon(); setSwipe(null); return; }
const index = clamp(direction > 0 ? Math.floor(next + 1 - SWIPE_COMMIT) : direction < 0 ? Math.ceil(next - 1 + SWIPE_COMMIT) : Math.round(next));
show(props.order[index]!);
gesture.current = null;
const ms = settleMs(Math.abs(index - (live.current ?? next)));
clearTimeout(settled.current);
setSettling(ms);
settled.current = setTimeout(() => setSettling(null), ms);
live.current = null;
setSwipe(null);
};
// A swipe along either axis moves the strip with it, so a mouse wheel works as well as a trackpad; it has ended when
// its events stop.
const onWheel = (event: WheelEvent) => {
// A pinch arrives as a wheel with Control held; it zooms the page, not the strip.
if (event.ctrlKey) return;
const delta = Math.abs(event.deltaX) > Math.abs(event.deltaY) ? event.deltaX : event.deltaY;
const pixels = event.deltaMode === 1 ? delta * 16 : event.deltaMode === 2 ? delta * stage.width : delta;
if (!pixels) return;
const next = clamp(begin() + pixels / pitch);
// The way the whole swipe went, not its last event: momentum's last 1px back must not turn a swipe around.
const direction = Math.sign(next - gesture.current!.start) || Math.sign(pixels);
moveTo(next, direction);
clearTimeout(swipeEnd.current);
swipeEnd.current = setTimeout(() => settle(next, direction), SWIPE_END_MS);
};
// A finger drags the strip under it; on lifting, a flick carries it on by its speed before it settles. A mouse
// keeps its clicks and wheel.
const onPointerDown = (event: PointerEvent) => {
// A browser that sent no click after the last swipe must not have this press's click swallowed, by finger or mouse.
swallowClick.current = false;
// A touch whose end never reached the strip (it slid off before it moved enough to be held) gives way to this one.
if (event.pointerType === "mouse" || touch.current?.id === event.pointerId) return;
touch.current = { id: event.pointerId, x: event.clientX, from: begin(), lastX: event.clientX, lastAt: event.timeStamp, velocity: 0, moved: false };
};
const onPointerMove = (event: PointerEvent) => {
const current = touch.current;
if (!current || current.id !== event.pointerId) return;
const dx = event.clientX - current.x;
if (!current.moved && Math.abs(dx) < TAP_SLOP_PX) return;
// A wheel swipe's settle waits until the finger takes over; a tap leaves it to settle.
if (!current.moved) { current.moved = true; clearTimeout(swipeEnd.current); event.currentTarget.setPointerCapture(event.pointerId); }
const elapsed = event.timeStamp - current.lastAt;
if (elapsed > 0) current.velocity = (event.clientX - current.lastX) / elapsed;
current.lastX = event.clientX; current.lastAt = event.timeStamp;
moveTo(clamp(current.from - dx / pitch), -Math.sign(dx));
};
const onPointerEnd = (event: PointerEvent) => {
const current = touch.current;
if (!current || current.id !== event.pointerId) return;
touch.current = null;
if (!current.moved) return;
swallowClick.current = true;
// A finger that stopped before lifting carries nothing on.
const velocity = event.timeStamp - current.lastAt > 80 ? 0 : current.velocity;
const next = clamp((live.current ?? position) - (velocity * FLICK_MS) / pitch);
settle(next, velocity ? -Math.sign(velocity) : -Math.sign(event.clientX - current.x));
};
const path = (props.shape.ancestors.get(props.focusedId) ?? []).flatMap(key => rowLabels.get(key) ?? []);
const plain = props.index.plainTitles || isLaunchScreen(focused);
const dot = props.dot(props.focusedId);
const header = (
<div className="screens-strip-path">
{path.map((label, position) => (
<span key={position} className="screens-strip-crumb" title={label}>
<span className={`screens-strip-crumb-label${plain ? "" : " path"}`}>{label}</span>
<ChevronRight aria-hidden="true" />
</span>
))}
<span className="screens-strip-crumb current" title={labelOf(focused, props.index)}>
{dot ? <span className={`screens-tree-dot ${dot}`} aria-hidden="true" /> : null}
<span className={`screens-strip-crumb-label${plain ? "" : " path"}`}>{labelOf(focused, props.index)}</span>
</span>
</div>
);
return (
<section className={`screens-strip${props.phone ? " phone" : ""}`} aria-label="Filmstrip" onWheel={onWheel}
onPointerDown={onPointerDown} onPointerMove={onPointerMove} onPointerUp={onPointerEnd} onPointerCancel={onPointerEnd}
onClickCapture={(event) => { if (swallowClick.current) { swallowClick.current = false; event.preventDefault(); event.stopPropagation(); } }}>
<div className="screens-strip-path">
{path.map((label, position) => (
<span key={position} className="screens-strip-crumb" title={label}>
<span className={`screens-strip-crumb-label${plain ? "" : " path"}`}>{label}</span>
<ChevronRight aria-hidden="true" />
</span>
))}
<span className="screens-strip-crumb current" title={labelOf(focused, props.index)}>
{dot ? <span className={`screens-tree-dot ${dot}`} aria-hidden="true" /> : null}
<span className={`screens-strip-crumb-label${plain ? "" : " path"}`}>{labelOf(focused, props.index)}</span>
</span>
</div>
<div ref={stageRef} className={`screens-strip-stage${swiping ? " swiping" : settling ? " settling" : ""}`}
style={settling ? ({ "--strip-settle-ms": `${settling}ms` } as CSSProperties) : undefined}>
{stage.width > 0 ? frames.map(({ id, screen, distance, x, scale, opacity, width, height }) => {
const artifact = primaryScreenshotForScreen(screen, props.index);
const url = artifactUrl(artifact);
const size = screenshotSize(artifact);
const placeholder = <span className="screens-strip-placeholder" aria-hidden="true" />;
const frameDiff = props.diff?.screens.get(id);
const label = labelOf(screen, props.index);
const middle = id === props.focusedId;
const hidden = opacity === 0;
return (
<div key={id} id={props.frameId(id)} data-screen-id={id}
className={`screens-strip-frame${middle ? " focused" : ""}${frameDiff ? ` ${diffClass(frameDiff)}` : ""}${props.dimmed(id) ? " dimmed" : ""}`}
style={{
width,
height,
opacity,
zIndex: Math.round((SCALES.length - distance) * 10),
transform: `translate(-50%, -50%) translateX(${x}px) scale(${scale})`,
}}>
{/* Only the middle frame is in the tab order: the arrows step the strip, and focus on a neighbour the stage
clips would scroll it off its centre. */}
<button type="button" className="screens-strip-target" aria-label={label} title={middle ? undefined : label}
tabIndex={middle ? undefined : -1} aria-hidden={hidden || undefined} disabled={hidden}
aria-pressed={props.selectedId === id} aria-current={middle ? "true" : undefined}
onClick={() => (middle ? props.onOpen(id) : props.onPick(id))}>
{/* A downloaded report's screenshot is a reference that ScreenshotImage resolves. */}
{url ? (
<ScreenshotImage src={url} alt="" draggable={false} decoding="async" loading={distance <= 1.5 ? "eager" : "lazy"}
{...(size ? { width: size.imageWidth, height: size.imageHeight } : {})} fallback={placeholder} />
) : placeholder}
</button>
</div>
);
}) : null}
</div>
</section>
<Filmstrip items={items} focusedId={props.focusedId} header={header} ariaLabel="Filmstrip" phone={props.phone}
frameId={props.frameId} idAttribute="data-screen-id"
frameClass={(id) => {
const frameDiff = props.diff?.screens.get(id);
return [frameDiff ? diffClass(frameDiff) : "", props.dimmed(id) ? "dimmed" : ""].filter(Boolean).join(" ") || undefined;
}}
label={(id) => { const screen = props.index.screens.get(id); return screen ? labelOf(screen, props.index) : id; }}
selected={(id) => props.selectedId === id}
onPick={props.onPick} onOpen={props.onOpen}
content={(id, { distance }) => {
const screen = props.index.screens.get(id)!;
const artifact = primaryScreenshotForScreen(screen, props.index);
const url = artifactUrl(artifact);
const size = screenshotSize(artifact);
const placeholder = <span className="screens-strip-placeholder" aria-hidden="true" />;
// A downloaded report's screenshot is a reference that ScreenshotImage resolves.
return url ? (
<ScreenshotImage src={url} alt="" draggable={false} decoding="async" loading={distance <= 1.5 ? "eager" : "lazy"}
{...(size ? { width: size.imageWidth, height: size.imageHeight } : {})} fallback={placeholder} />
) : placeholder;
}} />
);
}
+1 -1
View File
@@ -1,7 +1,7 @@
import type { CanvasView } from "./navigation";
/** Matches the left-to-right order of the persistent canvas toolbar. */
export const CANVAS_VIEW_ORDER = ["map", "screens", "compare"] as const satisfies readonly CanvasView[];
export const CANVAS_VIEW_ORDER = ["map", "screens", "compare", "flows"] as const satisfies readonly CanvasView[];
export type CanvasViewTransitionDirection = "forward" | "backward";
+5 -1
View File
@@ -13,6 +13,8 @@ export type CommandContext = {
selectedScreenId?: string | undefined;
/** The Screens page has a find field (desktop). */
find: boolean;
/** The report compares two runs of a flows file, so it has a Flows view. */
flows?: boolean;
/** The report can be handed off to an agent (a hosted preview). */
handoff: boolean;
pullRequest?: { origin?: "local" | "compare" | undefined; number?: number | undefined; url?: string | undefined } | undefined;
@@ -71,6 +73,7 @@ export function commandItems(context: CommandContext, actions: CommandActions):
if (context.canvas !== "map") add("go:map", "Go to", "Map", () => actions.selectCanvas("map"), { keywords: ["canvas", "overview"], shortcut: key(SHORTCUTS.canvas.map) });
if (context.canvas !== "screens") add("go:screens", "Go to", "Screens", () => actions.selectCanvas("screens"), { keywords: ["inventory", "grid", "list"], shortcut: key(SHORTCUTS.canvas.screens) });
if (hasChanges && context.canvas !== "compare") add("go:compare", "Go to", "Compare", () => actions.selectCanvas("compare"), { keywords: ["review", "changes", "diff", "before", "after"], shortcut: key(SHORTCUTS.canvas.compare) });
if (context.flows && context.canvas !== "flows") add("go:flows", "Go to", "Flows", () => actions.selectCanvas("flows"), { keywords: ["steps", "checkpoints", "journeys"], shortcut: key(SHORTCUTS.canvas.flows) });
if (context.find) add("go:find", "Go to", "Find a screen", actions.openFind, { keywords: ["filter", "screens", "search"], shortcut: "/" });
if (context.handoff) add("go:handoff", "Go to", "Hand off to AI", actions.openHandoff, { keywords: ["agent", "prompt", "claude", "codex", "copy"], shortcut: key(SHORTCUTS.app.handoff) });
if (!context.phone) add("go:shortcuts", "Go to", "Keyboard shortcuts", actions.openShortcuts, { keywords: ["keys", "help"], shortcut: "?" });
@@ -134,9 +137,10 @@ export function commandItems(context: CommandContext, actions: CommandActions):
/** The commands the empty palette leads with: what a reader on this canvas is most likely to do next. */
export function suggestedCommands(items: readonly SearchItem[], canvas: CanvasView, limit = 6): SearchItem[] {
const priority: Record<CanvasView, string[]> = {
map: ["changes:next", "changes:open-selected", "go:compare", "camera:fit", "go:find", "go:handoff", "go:screens", "view:inspector"],
map: ["go:flows", "changes:next", "changes:open-selected", "go:compare", "camera:fit", "go:find", "go:handoff", "go:screens", "view:inspector"],
screens: ["go:find", "changes:next", "changes:open-selected", "go:compare", "go:map", "go:handoff", "view:inspector"],
compare: ["changes:next", "changes:previous", "go:map", "go:screens", "go:handoff", "go:shortcuts"],
flows: ["go:map", "go:screens", "go:handoff", "go:shortcuts"],
};
return priority[canvas].flatMap((id) => items.filter((item) => item.id === `command:${id}`)).slice(0, limit);
}
+3 -2
View File
@@ -42,8 +42,9 @@ export function releaseEmbeddedArtifactUrl(reference: string): boolean {
export function viewerLocation(): Pick<Location, "pathname" | "search" | "hash"> {
if (!activeArtifactReport) return window.location;
// A report link opens on the Map, or in Compare when it names one changed screen (`view=compare`).
const view = new URLSearchParams(window.location.search).get("view") === "compare" ? "/compare" : "/map";
// A report link opens on the Map, on Flows for a flow review, or in Compare when it names one changed screen (`view=compare`).
const view = new URLSearchParams(window.location.search).get("view") === "compare" ? "/compare"
: activeArtifactReport.data.exploration?.flowReview ? "/flows" : "/map";
const url = new URL(window.location.hash.slice(1) || `${view}${window.location.search}`, "https://report.invalid");
return { pathname: url.pathname, search: url.search, hash: "" };
}
+156
View File
@@ -0,0 +1,156 @@
import type { FlowReview, FlowReviewCheckpoint, FlowReviewFlow, FlowReviewSide } from "../../../src/contracts/flow-comparison";
import type { ExplorationFlowStep } from "../../../src/contracts/standalone-report";
/** Each kind of checkpoint and flow the Flows view tells apart, by an icon of its own shape and a word. */
export type FlowMark = "changed" | "unchanged" | "stopped" | "not-reached" | "added" | "removed" | "not-comparable";
export const FLOW_MARK_LABELS: Record<FlowMark, string> = {
changed: "Changed",
unchanged: "Unchanged",
stopped: "Stopped",
"not-reached": "Not reached",
added: "Added",
removed: "Removed",
"not-comparable": "Different size",
};
export type FlowFilter = "changed" | "all";
export type FlowSide = "before" | "after";
const hasPicture = (side: FlowReviewSide | undefined): side is { screenshot: string; text?: string[] } => side !== undefined && "screenshot" in side;
/** A checkpoint's mark: what the comparison found, and for a missing picture, which build never reached it. */
export function checkpointMark(checkpoint: FlowReviewCheckpoint): FlowMark {
switch (checkpoint.verdict) {
case "changed": return "changed";
case "unchanged": return "unchanged";
case "added": return "added";
case "removed": return "removed";
case "different-size": return "not-comparable";
case "missing": return "not-reached";
}
}
/** The word under a checkpoint: a missing picture says which build has none. */
export function checkpointLabel(checkpoint: FlowReviewCheckpoint): string {
if (checkpoint.verdict !== "missing") return FLOW_MARK_LABELS[checkpointMark(checkpoint)];
return hasPicture(checkpoint.after) ? "Not reached before" : "Not reached";
}
/** A run that ended the flow early: it failed at a step, or the run stopped during or before it. */
export function stoppedEarly(flow: FlowReviewFlow): boolean {
return flow.after !== undefined && flow.after.status !== "completed";
}
/** Whether the Changed filter keeps a flow: any checkpoint the captures do not show unchanged, or a run that stopped. */
export function flowNeedsReview(flow: FlowReviewFlow): boolean {
return stoppedEarly(flow) || flow.checkpoints.some(checkpoint => checkpoint.verdict !== "unchanged");
}
/** A flow's mark: a stop first, then a flow only one run has, then its most pressing checkpoint. */
export function flowMark(flow: FlowReviewFlow): FlowMark {
if (stoppedEarly(flow)) return "stopped";
if (!flow.after) return "removed";
if (!flow.before) return "added";
const marks = new Set(flow.checkpoints.map(checkpointMark));
if (marks.has("changed") || marks.has("not-comparable")) return "changed";
return (["not-reached", "added", "removed"] as const).find(mark => marks.has(mark)) ?? "unchanged";
}
/** One line under a flow's name: whether it stopped, then how many checkpoints of each kind, the most pressing first. */
export function flowStatusLine(flow: FlowReviewFlow): string {
const counts = new Map<FlowMark, number>();
for (const checkpoint of flow.checkpoints) {
const mark = checkpointMark(checkpoint);
counts.set(mark, (counts.get(mark) ?? 0) + 1);
}
const parts = (["not-reached", "changed", "not-comparable", "added", "removed"] as const)
.filter(mark => counts.get(mark))
.map(mark => `${counts.get(mark)} ${FLOW_MARK_LABELS[mark].toLowerCase()}`);
if (stoppedEarly(flow)) return ["Stopped", ...parts].join(" · ");
return parts.length ? parts.join(" · ") : "Unchanged";
}
export function visibleFlows(review: FlowReview, filter: FlowFilter): FlowReviewFlow[] {
return filter === "all" ? review.flows : review.flows.filter(flowNeedsReview);
}
/** A flow's problems: every checkpoint the captures do not show unchanged, in the flow's order. */
export function problemIndexes(flow: FlowReviewFlow): number[] {
return flow.checkpoints.flatMap((checkpoint, index) => checkpoint.verdict === "unchanged" ? [] : [index]);
}
/** Where a flow opens: its first problem, or its first checkpoint when nothing changed. */
export function firstProblem(flow: FlowReviewFlow): number {
return problemIndexes(flow)[0] ?? 0;
}
/** The flow's steps as the after run took them, or the before run's when the after run has none recorded. */
export function flowSteps(flow: FlowReviewFlow): { steps: ExplorationFlowStep[]; failedStep?: number; side: FlowSide } | undefined {
if (flow.after?.steps) return { steps: flow.after.steps, ...(flow.after.failedStep !== undefined ? { failedStep: flow.after.failedStep } : {}), side: "after" };
if (flow.before?.steps) return { steps: flow.before.steps, side: "before" };
return undefined;
}
export type StepBefore = { step: Exclude<ExplorationFlowStep, { checkpoint: string }>; failed: boolean; ran: boolean };
/**
* The steps between the previous checkpoint and this one, as the flows file names them, with the one the after run
* failed at and the ones it never ran. Checkpoints match by name, as the comparison pairs them.
*/
export function stepsBefore(flow: FlowReviewFlow, index: number): StepBefore[] {
const recorded = flowSteps(flow);
const checkpoint = flow.checkpoints[index];
if (!recorded || !checkpoint) return [];
const key = (name: string) => name.trim().toLowerCase();
const at = recorded.steps.findIndex(step => "checkpoint" in step && key(step.checkpoint) === key(checkpoint.name));
if (at < 0) return [];
let start = at;
while (start > 0 && !("checkpoint" in recorded.steps[start - 1]!)) start -= 1;
const failed = recorded.side === "after" ? recorded.failedStep : undefined;
return recorded.steps.slice(start, at).flatMap((step, offset) => "checkpoint" in step ? [] : [{
step,
failed: failed === start + offset + 1,
ran: failed === undefined || start + offset + 1 <= failed,
}]);
}
/**
* What text the checkpoint's screen showed in one build and not the other, in each build's order. A screen without its
* text recorded on either side has none to compare.
*/
export function textChanges(checkpoint: FlowReviewCheckpoint): { removed: string[]; added: string[] } | undefined {
const before = hasPicture(checkpoint.before) ? checkpoint.before.text : undefined;
const after = hasPicture(checkpoint.after) ? checkpoint.after.text : undefined;
if (!before || !after) return undefined;
const was = new Set(before), now = new Set(after);
return { removed: before.filter(line => !now.has(line)), added: after.filter(line => !was.has(line)) };
}
/** A changed area no taller than a line or two of text, in percent of the screenshot's height. */
const TEXT_SIZED_AREA = 3;
/**
* The text a checkpoint changed, in one line, where the picture cannot carry it: when every changed area is text-sized,
* too small to read at the size the Viewer shows the screen. A larger change is read off the picture.
*/
export function textChangeLine(checkpoint: FlowReviewCheckpoint): string | undefined {
const changes = textChanges(checkpoint);
const regions = checkpoint.regions ?? [];
if (!changes || !regions.length || regions.some(region => region.height > TEXT_SIZED_AREA)) return undefined;
const { removed, added } = changes;
const more = (lines: string[]) => lines.length > 1 ? ` and ${lines.length - 1} more` : "";
if (removed.length === 1 && added.length === 1) return `“${added[0]}”, was “${removed[0]}”`;
if (added.length && !removed.length) return `Adds “${added[0]}”${more(added)}`;
if (removed.length && !added.length) return `Drops “${removed[0]}”${more(removed)}`;
if (added.length) return `“${added[0]}”${more(added)}, was “${removed[0]}”${more(removed)}`;
return undefined;
}
/** A side's screenshot, as the asset's SHA-256 the report keeps it under, or why it has none. */
export function sidePicture(side: FlowReviewSide | undefined): { sha256: string } | { missing: string } | undefined {
if (!side) return undefined;
if (!hasPicture(side)) return { missing: side.missing };
const match = /([a-f0-9]{64})\.(png|jpg)$/.exec(side.screenshot);
return match ? { sha256: match[1]! } : { missing: "The report has no screenshot for this checkpoint." };
}
+10 -5
View File
@@ -3,7 +3,7 @@
// and the URL is that route written out, Figma-style: a path for the place and a query for the
// selection. This module is DOM-free on purpose so the grammar can be tested in plain node.
/** Map and Screens are canvases; Compare is the Before/After review of one changed screen. */
export type CanvasView = "map" | "screens" | "compare";
export type CanvasView = "map" | "screens" | "compare" | "flows";
export type ThemePreference = "light" | "dark" | "system";
export type MapDirection = "TB" | "LR";
/** The explicitly selected amount of Map hierarchy, independent from camera zoom. */
@@ -44,6 +44,8 @@ export type Route = {
/** What the grammar needs from the app: which baselines exist, and which screens record a route. */
export type RouteContext = {
defaultBaseline: string;
/** The report compares two runs of a flows file: Flows is one of its views, and the one a link that names none opens on. */
flows?: boolean;
baselines: readonly string[];
/** Screen id → normalised expo-router route, for the screens that record one. */
screenRoutes: ReadonlyMap<string, string>;
@@ -85,7 +87,7 @@ export type FocusRequest = {
preserveMapContent?: boolean;
};
const CANVASES: CanvasView[] = ["map", "screens", "compare"];
const CANVASES: CanvasView[] = ["map", "screens", "compare", "flows"];
const INSPECTS: InspectView[] = ["session", "diff"];
const DIRECTIONS: TransitionDirection[] = ["in", "out"];
const FOCUSES: FocusStatus[] = ["added", "modified", "accessibility-only", "removed", "needs-review", "source-affected"];
@@ -166,7 +168,7 @@ function routeFitsPath(route: string | undefined): route is string {
return route !== undefined && route !== "/";
}
/** `#workspace/<canvas>` and the bare `#screens` predate path routing and are still linked; a retired canvas (flows) lands on the map. */
/** `#workspace/<canvas>` and the bare `#screens` predate path routing and are still linked. */
function parseLegacyHash(hash: string): CanvasView | undefined {
const [first, second] = hash.replace(/^#/, "").split("/");
if (first === "screens" && !second) return "screens";
@@ -180,7 +182,8 @@ export function canonicalRoute(route: Route, ctx: RouteContext): Route {
// Graph was removed as a destination. Runtime history or callers compiled against an older
// route type can still hand us that value, so settle it to Map before deriving scope or URLs.
const legacyGraph = (route.canvas as string) === "graph";
const canvas = isCanvas(route.canvas) ? route.canvas : "map";
// Flows is a view only of a flow review; elsewhere its links (and the retired Flows canvas's) land on the map.
const canvas = isCanvas(route.canvas) && (route.canvas !== "flows" || ctx.flows) ? route.canvas : "map";
const baseline = route.baseline === "" || ctx.baselines.includes(route.baseline)
? route.baseline
: ctx.defaultBaseline;
@@ -259,7 +262,7 @@ export function sanitizeViewerRoute(route: Route): Route {
export function parseLocation(location: LocationLike, ctx: RouteContext): Route {
const params = new URLSearchParams(location.search);
let canvas: CanvasView = "map";
let canvas: CanvasView = ctx.flows ? "flows" : "map";
let legacyGraph = false;
let screenRoute: string | undefined;
@@ -344,6 +347,7 @@ export const CANVAS_LABELS: Record<CanvasView, string> = {
map: "Map",
screens: "Screens",
compare: "Compare",
flows: "Flows",
};
/** One line each, shown under the canvas name on hover, so the views are easy to tell apart. */
@@ -351,6 +355,7 @@ export const CANVAS_DESCRIPTIONS: Record<CanvasView, string> = {
map: "How screens connect",
screens: "Every screen in one grid",
compare: "Before and After of each changed screen",
flows: "Each flow, step by step",
};
export const FOCUS_LABELS: Record<FocusStatus, string> = {
+13 -2
View File
@@ -109,19 +109,21 @@ export const SHORTCUTS = {
map: plain("KeyM", "M"),
screens: plain("KeyS", "S"),
compare: plain("KeyC", "C"),
flows: plain("KeyP", "P"),
},
/** The canvases by their place in the toolbar, as tabs are numbered elsewhere. Apple keyboards only (CANVAS_NUMBER_KEYS). */
canvasByNumber: {
map: ctrl("Digit1", "1"),
screens: ctrl("Digit2", "2"),
compare: ctrl("Digit3", "3"),
flows: ctrl("Digit4", "4"),
},
/** Screens only, on desktop: G for the grid and E for the filmstrip, as Lightroom's Library switches Grid and Loupe. */
/** Screens only, on desktop: G for the grid and E for the filmstrip, as Lightroom's Library switches Grid and Loupe. Flows takes them for Row and Focus. */
screens: {
grid: plain("KeyG", "G"),
filmstrip: plain("KeyE", "E"),
},
/** Compare only. The arrows walk changed screens; J and K walk the changed areas of one. */
/** Compare only. The arrows walk changed screens; J and K walk the changed areas of one. Flows takes the arrows, J, K and X for its problems, areas and flip. */
compare: {
previousScreen: plain("ArrowLeft", "←"),
nextScreen: plain("ArrowRight", "→"),
@@ -142,6 +144,15 @@ export const SHORTCUTS = {
},
} as const;
/** Flows borrows Compare's keys: the arrows, J, K and X step and flip as there. */
export const FLOWS_KEYS = {
previous: SHORTCUTS.compare.previousScreen,
next: SHORTCUTS.compare.nextScreen,
nextArea: SHORTCUTS.compare.nextArea,
previousArea: SHORTCUTS.compare.previousArea,
flip: SHORTCUTS.compare.flipSide,
} as const;
/** The hint, in the idiom of the keyboard reading it: Apple's ⌥⇧⌘ order, or Ctrl+Alt+Shift. */
export function formatShortcut(shortcut: Shortcut, apple = APPLE): string {
if (apple) {