mirror of
https://github.com/expo/explore.git
synced 2026-11-15 13:18:16 -10:00
Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3f70e3d050 | ||
|
|
f620577e30 | ||
|
|
6c927b1f22 | ||
|
|
b72a70f7d8 | ||
|
|
fecf040ce5 | ||
|
|
e47399440f | ||
|
|
46b46bb184 | ||
|
|
942bdaad40 | ||
|
|
ed31b0c767 |
No files matched your search
+16
-5
@@ -14,7 +14,7 @@ Source of truth: `templates/explore-review.yml`, `templates/explore-baseline.yml
|
||||
| Pipeline | `capture:review` → `runCiReview` | The same `runCiReview`, called directly |
|
||||
| EAS credentials | `EXPO_TOKEN` secret | Local `eas login` session or `EXPO_TOKEN` |
|
||||
| Model credentials (agent mode) | `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` | Claude Code or Codex subscription login; API keys are removed from the subprocess |
|
||||
| Base cache | A baseline workflow artifact for the exact base SHA, when one exists | None; both sides are captured |
|
||||
| Base cache | A baseline workflow artifact for the exact base SHA, when one exists, or the baseline run still capturing it, waited for alongside head | None; both sides are captured |
|
||||
| Output | Report artifact + sticky PR comment with **Open** / **zip** | `runs/<id>/artifacts/report.zip` + `PR-comment.md`, opened with `ci:local:view` |
|
||||
|
||||
## 1. Installing Explore in an app repository
|
||||
@@ -49,8 +49,8 @@ Each workflow starts with a `targets` job: it installs the pinned CLI and runs `
|
||||
| | checkout `app-base`, `app-head` | Checks out the exact base and head SHAs with full history and without persisted credentials. |
|
||||
| | Verify deterministic source planning | `source:verify` builds the capture plan from source alone in two fresh processes and compares the bytes. Runs before any build or simulator work and needs no credentials. |
|
||||
| | Install app dependencies | `npm ci --ignore-scripts` in both checkouts (`APP_INSTALL_DIR` for monorepos). |
|
||||
| | Find / Download optional exact-base evidence | `ci:find-baseline` looks for a finished run's `explore-baseline-<base sha>-<target>` artifact (a run another target's baseline failed in counts: a baseline job uploads its artifact only after its own steps passed) (target `ios` also accepts the older unsuffixed `explore-baseline-<base sha>`) from the baseline workflow on the default branch and downloads it as `EXPLORE_CAPTURE_CACHE`. Missing evidence is not an error. |
|
||||
| | Capture review | `capture:review` (§3). `EXPO_TOKEN`, the provider key and the app's `{{secret:NAME}}` values reach only this step. The sticky PR comment is updated while it runs. It writes `preview=<hosting>` to the step outputs, which decides whether the deploy job runs. |
|
||||
| | Find / Download optional exact-base evidence | `ci:find-baseline` looks for a finished run's `explore-baseline-<base sha>-<target>` artifact (a run another target's baseline failed in counts: a baseline job uploads its artifact only after its own steps passed) (target `ios` also accepts the older unsuffixed `explore-baseline-<base sha>`) from the baseline workflow on the default branch and downloads it as `EXPLORE_CAPTURE_CACHE`. With none, it names a baseline run of the base commit still queued or capturing (`pending`), held to the same checks: the baseline workflow, the default branch of this repository, a `push` or `workflow_dispatch`, exactly the base SHA; nothing from the PR head selects it. Missing evidence and a failed lookup are not errors. |
|
||||
| | Capture review | `capture:review` (§3). With `EXPLORE_BASELINE_RUN` (the find step's `pending`) it waits for that run's baseline of this target (§3.1). `EXPO_TOKEN`, the provider key and the app's `{{secret:NAME}}` values reach only this step. The sticky PR comment is updated while it runs. It writes `preview=<hosting>` to the step outputs, which decides whether the deploy job runs. |
|
||||
| | Stop capture sessions and check the checkouts | `ci:after-capture`, whenever the PR was resolved, success or not: stops the EAS Simulator sessions named in `head-session` / `base-session`, nothing else, then fails with `CHECKOUT_MODIFIED` if capture modified an app checkout. |
|
||||
| | Upload captured review / diagnostics | `explore-capture-<run>-<attempt>-<target>` feeds the report job. On failure `explore-diagnostics-<run>-<attempt>-<target>` keeps plans, exports and agent diagnostics for 7 days. |
|
||||
| report (matrix: target) | Download the capture, install Explore | Runs without Expo or model credentials, after every capture job, whether or not each succeeded (`!cancelled()`). Each leg downloads its own target's capture by an exact-name pattern: a target whose capture failed uploaded none, so its leg packages and publishes nothing, and the other targets' reports still reach the comment. |
|
||||
@@ -68,7 +68,7 @@ Validate baseline source → Install Explore → Verify deterministic source pla
|
||||
|
||||
### 2.4 `explore-compare.yml`
|
||||
|
||||
The same jobs as a PR review, for any `base` and `head` without a pull request. The `targets` job resolves both refs once with `ci:compare-refs` (a tag wins over a branch of the same name; the same commit twice fails with `COMPARE_SAME_COMMIT`) and passes the comparison to the capture jobs, which run `capture:review --compare` with no GitHub requests. The optional `scope` input overrides `exploration.scope` for that run; `expires-in` is accepted and ignored. The report job's `ci:publish` writes each target's **Open** and **zip** links to the run summary, and the deploy job adds the hosted link there; nothing comments on a pull request.
|
||||
The same jobs as a PR review, for any `base` and `head` without a pull request. The `targets` job resolves both refs once with `ci:compare-refs` (a tag wins over a branch of the same name; the same commit twice fails with `COMPARE_SAME_COMMIT`) and passes the comparison to the capture jobs, which run `capture:review --compare`; its only GitHub requests are the wait for a baseline run still capturing the base (§3.1), with the job's `actions: read` token. The optional `scope` input overrides `exploration.scope` for that run; `expires-in` is accepted and ignored. The report job's `ci:publish` writes each target's **Open** and **zip** links to the run summary, and the deploy job adds the hosted link there; nothing comments on a pull request.
|
||||
|
||||
## 3. Inside `capture:review` (`runCiReview`)
|
||||
|
||||
@@ -77,7 +77,7 @@ The same jobs as a PR review, for any `base` and `head` without a pull request.
|
||||
| 1 | `plan` | `buildVerifiedPlan`: snapshot both sources, compute the two-dot diff, run `plan-worker` twice with reversed input order and compare the canonical plans byte for byte. Summarises `sourceChange` per surface into `execution-plan.json`; the PR comment shows the planned capture count and warnings. | `SOURCE_ANALYSIS_FAILED`, inconsistent plan |
|
||||
| 2 | `preflight` | Checks `EXPO_TOKEN`. `prepareCiExploration`: nothing for deterministic mode; for agent mode, confirms the instructions file is committed, creates the provider session and makes one short tool-free model request. | `AGENT_API_KEY_MISSING`, `AGENT_INSTRUCTIONS_UNCOMMITTED`, `AGENT_MODEL_NOT_FOUND` |
|
||||
| 3 | `capture-head` | `runCiMap(head)` (§4). Records head's runtime profile and build. | `CAPTURE_*`, `SETUP_*`, `EAS_*` |
|
||||
| 4 | `capture-base` | Tries the cache first: a baseline is reused only when its recipe matches (tools, config, instructions, device, profile, observer hash, runtime profile). Only routes missing from the cache run through `runCiMap(base)`. | same |
|
||||
| 4 | `capture-base` | Tries the cache first: a baseline is reused only when its recipe matches (tools, config, instructions, device, profile, observer hash, runtime profile). Only routes missing from the cache run through `runCiMap(base)`. With no cache but a baseline run still capturing the base, base starts only once the wait for it ends (§3.1). | same |
|
||||
| 5 | `compare` | `assessPublicEvidence` compares pixels and accessibility for route pairs verified on both sides; `prepareCaptureReview` assigns verdicts (§6). | |
|
||||
| 6 | `capture-validation` | Every selected route was captured on both sides. | `CAPTURE_INCOMPLETE` |
|
||||
| 7 | `comparison-validation` | Every pair was compared visually and for accessibility. | `COMPARISON_INCOMPLETE` |
|
||||
@@ -86,6 +86,17 @@ The same jobs as a PR review, for any `base` and `head` without a pull request.
|
||||
|
||||
A failure at any stage is recorded in `state.failure`, a partial report is written when a plan exists, and the PR comment switches to the failed state.
|
||||
|
||||
### 3.1 Waiting for a baseline still being captured
|
||||
|
||||
Right after a merge, the next pull request's review starts while the baseline workflow is still capturing the new base: in brentvatne/euxy's review run 39348956443 it had started 15 s before, and every target captured its base too (10–14 minutes per target instead of about 6, on twice the EAS Simulator sessions). A baseline job takes 6–8 minutes and head capture 5–8, so `capture:review` waits for the baseline beside head instead of before it (`src/ci/baseline-wait.ts`):
|
||||
|
||||
1. The wait starts after preflight, only when `EXPLORE_BASELINE_RUN` names a run, no downloaded baseline matches, base has routes to capture, and `capture.baselineWaitMinutes` (default 15, 0 never waits) is not 0. Head starts capturing at once.
|
||||
2. Every 20 s it reads the run (checked again by the same trust rules as `ci:find-baseline`), this target's artifact (`explore-baseline-<sha>-<target>`) in that run, and the run's jobs, logging `waiting for main's baseline run <id> of <sha> (<target>: in progress, 3 min)` once a minute or when the state changes. Each read retries GitHub's 5xx and network failures (`retryGitHubLookup`).
|
||||
3. When the artifact is listed, it is downloaded in-process (the token goes to GitHub, never to the storage URL it redirects to), unpacked into `<output>/baseline-download` within the cache's size limits, and restored as a downloaded baseline once head has its runtime profile. Another target's failed job does not matter; each target's job uploads its own artifact.
|
||||
4. The wait ends without a baseline when this target's job fails, is cancelled or is skipped, the run ends without listing the artifact for a minute, the time runs out, GitHub refuses the token (`actions: read` missing) or keeps failing, or head fails. Base then captures at once, beside head when the sides capture concurrently. The wait never fails the review.
|
||||
|
||||
Head capture is never delayed. When the baseline arrives while head captures, base costs only its restore; when it does not, base starts the moment the wait gives up, so a failed baseline delays base only by the time it took to fail. Sessions are planned as without a wait: with a share of two or more, head keeps its half while base waits, so base can start in the rest; with one session, base follows head as before, and the wait may go on past head, up to its limit.
|
||||
|
||||
## 4. Capturing one side (`runCiMap`)
|
||||
|
||||
| # | Stage | Work |
|
||||
|
||||
@@ -198,6 +198,41 @@ npx @expo/explore explore:session <session-id> --env-file path/.env.eas-simulato
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Take the same steps on every run and device</strong></summary>
|
||||
|
||||
A model picks its own path, so two runs (or an iPhone and an Android phone) can reach different screens. To get the same screens every time, write the steps down once, for example after a model run showed you the app, and pass them with `--replay`. No model login or API key is needed.
|
||||
|
||||
```sh
|
||||
npx @expo/explore explore:binary ./my-app.app --replay steps.json
|
||||
npx @expo/explore explore:session --replay steps.json # an iPhone Duo pose, or any session you set up by hand
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"steps": [
|
||||
{ "capture": "Home" },
|
||||
{ "action": "tap", "label": "Settings" },
|
||||
{ "capture": "Settings" },
|
||||
{ "action": "tap", "label": "Dark" },
|
||||
{ "capture": "Settings", "state": "dark" },
|
||||
{ "action": "tap", "label": "Learn", "platforms": ["ios"] },
|
||||
{ "capture": "Learn", "platforms": ["ios"] },
|
||||
{ "action": "tap", "label": "Got it", "optional": true }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- An action is one of `tap` or `fill` (with `label` or `identifier`), `type`, `scroll` (`direction`), `wait` (`milliseconds`), `open` (`url`), `back`, and `dismiss-keyboard`, as in the model's own tools. The same safety rules apply: destructive or app-leaving taps are refused. A step takes only the fields listed here; any other field (a typo such as `platform`) is an error.
|
||||
- `{ "capture": "Title" }` photographs the current screen under that title; add `"state"` for another state of a screen already captured. A capture of a screen the plan already captured under another title is reported as looking the same, and the screen keeps its first title.
|
||||
- `"platforms": ["ios"]` limits a step to those platforms, for screens one platform does not have. `"optional": true` skips an action that fails (a one-time dialog) instead of stopping; the summary lists the steps it skipped.
|
||||
- A `fill` or `type` text can be a `{{secret:NAME}}`, as in `--hints`: Explore types the value of the environment variable `NAME`, and a variable that is not set fails before a device starts.
|
||||
- If a required action fails, the run stops there and the report still shows what was captured, with a gap naming the step and the screens it did not reach.
|
||||
- The screen and action budgets still apply. A plan with more captures or actions than they allow is refused before a device starts, as is one with no steps for the platform; raise `--max-screens` and `--max-actions` for a long plan. If a budget still runs out (each screen the plan reaches also counts), the gap names the planned screens not reached.
|
||||
- `--provider`, `--model` and `--hints` only steer a model, so they cannot be combined with `--replay`.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>What a run without source shows</strong></summary>
|
||||
|
||||
@@ -457,7 +492,7 @@ jobs:
|
||||
- The run summary links each target's report (**Open** and **zip**, like a PR comment) and, with hosting, the hosted report.
|
||||
- `setup --workflows` installs it from 0.19.0; `npx @expo/explore@latest setup --workflows` adds it to an existing setup. The caller grants `contents: read` and `actions: read`; `secrets: inherit` passes `EXPO_TOKEN` and any app-test secrets.
|
||||
- Call it once per workflow run. A called workflow runs inside its caller's run, so a second call in the same run would reuse the first one's artifact names; use separate workflows instead.
|
||||
- When the baseline workflow captured `base` in the last 30 days, with the same config and Explore version, that capture is reused and only `head` is captured. The report artifact is kept 30 days; a hosted link lasts as long as its EAS Hosting deployment (an aliased one until the alias moves).
|
||||
- When the baseline workflow captured `base` in the last 30 days, with the same config and Explore version, that capture is reused and only `head` is captured. When it is still capturing `base` (right after a merge), the review waits for it while `head` captures, up to `capture.baselineWaitMinutes` (15 by default). The report artifact is kept 30 days; a hosted link lasts as long as its EAS Hosting deployment (an aliased one until the alias moves).
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
@@ -105,6 +105,7 @@ capture:
|
||||
maxActions: 500
|
||||
# maxSessions: 5 # the most EAS Simulator sessions one review runs at once, across all its targets and sides (default: 5, a typical plan's concurrent simulator limit). Each target's job gets floor(maxSessions / targets) sessions, at least one, and sessions left over go to foldable targets (pose): one target gets 5 (head and base at once, light and dark in their own sessions when that pays), two get 2 each, five get 1 each (head, then base). Below the number of targets, only that many target jobs run at once (the workflow's max-parallel, and a local review's targets), one session each. A side spreads its light and dark passes over separate sessions only when that saves it at least a minute, so the ceiling is a limit, not a target: a 3-route target stays on two sessions. euxy's four targets: 8 is two per target, 16 captures each appearance in its own session (measured: an iPhone side 6:12 → 4:26, one repack shared by its sessions, ~3.5 session-minutes more per side).
|
||||
# On a starter plan (one simulator per platform), set maxSessions to the plan's limit, e.g. 2 for an iOS and an Android target.
|
||||
# baselineWaitMinutes: 15 # a review that starts while the default branch's baseline run of its base commit is still capturing (right after a merge) waits up to this long for that target's baseline instead of capturing the base too (default 15; 0 never waits). Head captures meanwhile; base waits only where it would start, and captures itself if the baseline job fails or is cancelled, or the time runs out.
|
||||
repack: eas # eas (hosted workflow job; the default on EAS Simulator) | local (repack-app on the runner; the default with simulator: local)
|
||||
appearance: auto # auto (follow the app's userInterfaceStyle: automatic captures light and dark) | dark | light | [dark, light]
|
||||
launch: screenshot # screenshot (relaunch the app per appearance and keep its splash and first settled frame, default) | video (also keep a launch recording in the report) | off
|
||||
|
||||
@@ -111,6 +111,7 @@ capture:
|
||||
maxActions: 500
|
||||
# maxSessions: 5 # the most EAS Simulator sessions one review runs at once, across all its targets and sides (default: 5, a typical plan's concurrent simulator limit). Each target's job gets floor(maxSessions / targets) sessions, at least one, and sessions left over go to foldable targets (pose): one target gets 5 (head and base at once, light and dark in their own sessions when that pays), two get 2 each, five get 1 each (head, then base). Below the number of targets, only that many target jobs run at once (the workflow's max-parallel, and a local review's targets), one session each. A side spreads its light and dark passes over separate sessions only when that saves it at least a minute, so the ceiling is a limit, not a target: a 3-route target stays on two sessions. euxy's four targets: 8 is two per target, 16 captures each appearance in its own session (measured: an iPhone side 6:12 → 4:26, one repack shared by its sessions, ~3.5 session-minutes more per side).
|
||||
# On a starter plan (one simulator per platform), set maxSessions to the plan's limit, e.g. 2 for an iOS and an Android target.
|
||||
# baselineWaitMinutes: 15 # a review that starts while the default branch's baseline run of its base commit is still capturing (right after a merge) waits up to this long for that target's baseline instead of capturing the base too (default 15; 0 never waits). Head captures meanwhile; base waits only where it would start, and captures itself if the baseline job fails or is cancelled, or the time runs out.
|
||||
repack: eas # eas (hosted EAS workflow job, the default on EAS Simulator) | local (repack-app on the runner, the default with simulator: local; --repack overrides it for one run)
|
||||
appearance: auto # auto (follow the app's userInterfaceStyle: automatic captures light and dark) | dark | light | [dark, light]
|
||||
launch: screenshot # screenshot (relaunch the app per appearance and keep its splash and first settled frame, default) | video (also keep a launch recording in the report) | off
|
||||
@@ -391,7 +392,7 @@ The review workflow resolves the current PR through GitHub, checks out its exact
|
||||
| Review packaging | The bundle contains metadata and assets, not a standalone HTML page |
|
||||
| Publication | The artifact is uploaded before current-PR identity is checked again and the **Open** and **zip** links are posted |
|
||||
|
||||
The baseline workflow runs on default-branch pushes or manual default-branch dispatch. It captures one complete commit without needing a PR. Reports and baselines are retained for 30 days; intermediate evidence and diagnostics for 7 days.
|
||||
The baseline workflow runs on default-branch pushes or manual default-branch dispatch. It captures one complete commit without needing a PR. A review that starts while the baseline of its base commit is still being captured, as the next pull request's review does right after a merge, waits for it (`capture.baselineWaitMinutes`, 15 minutes by default) while head captures, instead of capturing the base too. Reports and baselines are retained for 30 days; intermediate evidence and diagnostics for 7 days.
|
||||
|
||||
`exploration.scope: changes` includes source-affected routes and the routes one navigation step before or after them, so a reviewer sees how each change is reached and where it leads. The context is not transitive: in most apps tabs, guards and Home connect every screen, so all ancestors and descendants would be the whole app. A layout or screen guard that sends every signed-out screen to sign-in is not a step. Forwarding routes have no screenshot of their own, so their redirect chains are followed to the first screen. Unresolved navigation still selects every route. `all` captures the full inventory.
|
||||
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@expo/explore",
|
||||
"version": "0.20.1",
|
||||
"version": "0.20.5",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@expo/explore",
|
||||
"version": "0.20.1",
|
||||
"version": "0.20.5",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@expo/repack-app": "0.10.1",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@expo/explore",
|
||||
"version": "0.20.1",
|
||||
"version": "0.20.5",
|
||||
"description": "Source-driven mobile app maps, deterministic CI capture, and visual reviews",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -202,6 +202,15 @@
|
||||
"mimeType": { "type": "string", "minLength": 1 },
|
||||
"width": { "type": "integer", "minimum": 0 },
|
||||
"height": { "type": "integer", "minimum": 0 },
|
||||
"turned": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["width", "height"],
|
||||
"properties": {
|
||||
"width": { "type": "number", "exclusiveMinimum": 0 },
|
||||
"height": { "type": "number", "exclusiveMinimum": 0 }
|
||||
}
|
||||
},
|
||||
"createdAt": { "$ref": "#/$defs/timestamp" }
|
||||
}
|
||||
},
|
||||
|
||||
@@ -33,6 +33,10 @@ Platform, bundle id, app name and version are read from the archive; do not pass
|
||||
|
||||
Budgets default to 15 minutes, 40 screens, 150 actions (`--max-duration-minutes`, `--max-screens`, `--max-actions`). Steer the model with `--hints "focus on the onboarding and settings"`, skip onboarding with `--launch-url myapp://home`, and allow a specific risky control with `--allow-destructive "Clear cache"`.
|
||||
|
||||
## The same steps on every run: `--replay`
|
||||
|
||||
To compare runs, or an iPhone with an Android phone, screen for screen, pass `--replay steps.json` to `explore:binary` or `explore:session`: Explore takes the steps in the file instead of a model's, and no model access is needed. The format is in the README ("Take the same steps on every run and device"): `{ "steps": [{ "capture": "Home" }, { "action": "tap", "label": "Settings" }, { "capture": "Settings" }] }`, with `platforms` to limit a step and `optional` for an action that may fail. The plan is checked before a device starts: unknown fields, a plan with no steps for the platform, more captures or actions than `--max-screens` / `--max-actions`, and a `{{secret:NAME}}` in a `fill` or `type` whose variable is not set are all refused (`REPLAY_PLAN_INVALID`, `REPLAY_PLAN_EMPTY`, `REPLAY_PLAN_OVER_BUDGET`, `EXPLORATION_SECRET_MISSING`). `--provider`, `--model` and `--hints` cannot be combined with it (`REPLAY_OPTIONS_IGNORED`).
|
||||
|
||||
## Sign-in and other app state
|
||||
|
||||
Without help, the model treats a sign-in wall as the end of the road: it captures it and stops (`blocked`). Before running, read what you can about the app (its source if you have it, its first screens otherwise) and decide what it needs.
|
||||
|
||||
+59
-27
@@ -7,6 +7,7 @@ import type { CaptureSetupAction } from "../config/explore-config.js";
|
||||
import { createExplorationInputs, explorationSecretNames } from "../config/exploration-inputs.js";
|
||||
import type { ExploreStore, PreparedArtifact } from "../map/store.js";
|
||||
import { CiError } from "../shared/ci-errors.js";
|
||||
import { turnedClockwise } from "../shared/png-evidence.js";
|
||||
import type { ActionKind, ActionRisk, Artifact, Controller, ElementRole, ExploreMap, ObservedElement, Observation, Screen } from "../types.js";
|
||||
|
||||
/** One look at the device: accessibility always, a PNG when the host could take a safe one. */
|
||||
@@ -21,13 +22,15 @@ export type BlackBoxFrame = {
|
||||
imageUnavailable?: { code: string; message: string };
|
||||
/** The foreground app differed from the explored one; the host already brought it back. */
|
||||
leftApp?: boolean;
|
||||
/** The image is on a portrait panel's axes under a landscape UI (an opened iPhone Duo), as the tree is; the report stores it turned upright. */
|
||||
landscape?: boolean;
|
||||
};
|
||||
|
||||
export type BlackBoxAction = CaptureSetupAction | { action: "back" } | { action: "dismiss-keyboard" } | { action: "type"; text: string }
|
||||
| { action: "tap"; frameId: string; x: number; y: number };
|
||||
|
||||
export type BlackBoxExplorerOptions = {
|
||||
session: CiAgentSession;
|
||||
session: Pick<CiAgentSession, "run">;
|
||||
store: ExploreStore;
|
||||
map: ExploreMap;
|
||||
app: { name: string; bundleId: string; platform: "ios" | "android"; version?: string };
|
||||
@@ -49,6 +52,8 @@ export type BlackBoxExplorerOptions = {
|
||||
appIcon?: Buffer;
|
||||
/** App-test inputs the operator hints reference as {{secret:NAME}}: the model and the map only ever hold the reference. */
|
||||
inputs?: ReturnType<typeof createExplorationInputs>;
|
||||
/** A sentence a budget stop's gap adds to its own, read once the session has stopped: a replay names the planned screens it did not reach. */
|
||||
budgetNote?: () => string | undefined;
|
||||
};
|
||||
|
||||
export type BlackBoxExplorerResult = {
|
||||
@@ -67,6 +72,12 @@ const EXTERNAL = /\b(open in|safari|chrome|browser|app store|play store|share|ma
|
||||
const VOLATILE = /\d|[:%$€£]|\b(am|pm)\b/gi;
|
||||
const INTERACTIVE = new Set(["button", "link", "cell", "switch", "tab", "menuitem", "textfield", "securetextfield", "searchfield", "textview", "edittext", "slider", "checkbox", "radiobutton", "segmentedcontrol", "toggle"]);
|
||||
|
||||
/** What the tools accept: their schemas, the checks on each call, and a replay plan checked before any device starts share these. */
|
||||
export const TOOL_LIMITS = { selector: 512, text: 2000, url: 2000, waitMs: 15_000, title: 80, state: 80, description: 500, summary: 4000, reason: 1000 } as const;
|
||||
|
||||
/** A screen seen after an action but never named keeps this title until the run ends and names it after the path to it. */
|
||||
export const isPlaceholderTitle = (title: string): boolean => /^Screen \d+$/.test(title);
|
||||
|
||||
const selectorProperties = {
|
||||
label: { type: "string", description: "Exact visible label of the target node, copied from the latest observation." },
|
||||
identifier: { type: "string", description: "Exact accessibility identifier of the target node, when it has one." },
|
||||
@@ -81,22 +92,22 @@ const tools: CiAgentTool[] = [
|
||||
inputSchema: { type: "object", properties: {
|
||||
action: { type: "string", enum: ["tap", "fill", "type", "scroll", "wait", "back", "dismiss-keyboard", "open"] },
|
||||
...selectorProperties,
|
||||
text: { type: "string", maxLength: 2000 }, url: { type: "string", maxLength: 2000 },
|
||||
text: { type: "string", maxLength: TOOL_LIMITS.text }, url: { type: "string", maxLength: TOOL_LIMITS.url },
|
||||
direction: { type: "string", enum: ["up", "down", "left", "right", "top", "bottom"] },
|
||||
milliseconds: { type: "integer", minimum: 1, maximum: 15_000 },
|
||||
milliseconds: { type: "integer", minimum: 1, maximum: TOOL_LIMITS.waitMs },
|
||||
frameId: { type: "string", description: "For a coordinate tap only: the latest observed frame." },
|
||||
x: { type: "number", minimum: 0, exclusiveMaximum: 1 }, y: { type: "number", minimum: 0, exclusiveMaximum: 1 },
|
||||
target: { type: "string", minLength: 1, maxLength: 80, description: "For a coordinate tap: what is being tapped, as a user would name it (\"search icon in the header\", \"first manga cover\")." },
|
||||
}, required: ["action"], additionalProperties: false } },
|
||||
{ name: "capture", description: "Name the latest observed frame (observe with includeImage) as a screen of the map, with a short human title. A screen is photographed the moment it first appears; capture gives it its title, and with state records another notable state of it.",
|
||||
inputSchema: { type: "object", properties: {
|
||||
frameId: { type: "string" }, title: { type: "string", minLength: 1, maxLength: 80 },
|
||||
description: { type: "string", maxLength: 500 }, state: { type: "string", maxLength: 80, description: "A variant of an already captured screen, such as 'empty' or 'menu open'." },
|
||||
frameId: { type: "string" }, title: { type: "string", minLength: 1, maxLength: TOOL_LIMITS.title },
|
||||
description: { type: "string", maxLength: TOOL_LIMITS.description }, state: { type: "string", maxLength: TOOL_LIMITS.state, description: "A variant of an already captured screen, such as 'empty' or 'menu open'." },
|
||||
}, required: ["frameId", "title"], additionalProperties: false } },
|
||||
{ name: "finish", description: "End the exploration when every reachable screen has been captured or the budget is nearly used. Include a summary of what the app does and what was not reached.",
|
||||
inputSchema: { type: "object", properties: { summary: { type: "string", minLength: 1, maxLength: 4000 } }, required: ["summary"], additionalProperties: false } },
|
||||
inputSchema: { type: "object", properties: { summary: { type: "string", minLength: 1, maxLength: TOOL_LIMITS.summary } }, required: ["summary"], additionalProperties: false } },
|
||||
{ name: "stop_blocked", description: "Stop because the app cannot be explored further (login wall without credentials, crash loop, permission prompt that cannot be dismissed). Keeps the evidence so far.",
|
||||
inputSchema: { type: "object", properties: { reason: { type: "string", minLength: 1, maxLength: 1000 } }, required: ["reason"], additionalProperties: false } },
|
||||
inputSchema: { type: "object", properties: { reason: { type: "string", minLength: 1, maxLength: TOOL_LIMITS.reason } }, required: ["reason"], additionalProperties: false } },
|
||||
];
|
||||
|
||||
const INSTRUCTIONS = `You are exploring a prebuilt mobile app on a remote simulator through accessibility and screenshots. Nobody gave you its source. Your job is to produce a map of its screens: visit every reachable screen, capture each one once with a clear title, and try each distinct kind of control so transitions are recorded.
|
||||
@@ -412,7 +423,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
}
|
||||
|
||||
/** Seen after an action but never named by the model (its screenshot was taken on arrival). */
|
||||
const isPlaceholder = (screen: Screen): boolean => /^Screen \d+$/.test(screen.title);
|
||||
const isPlaceholder = (screen: Screen): boolean => isPlaceholderTitle(screen.title);
|
||||
/** Fold an uncaptured placeholder into the screen the model says it is. */
|
||||
const mergeScreens = (from: Screen, into: Screen): void => {
|
||||
for (const item of map.actionAttempts) {
|
||||
@@ -437,8 +448,22 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
if (currentScreenId === from.id) currentScreenId = into.id;
|
||||
};
|
||||
|
||||
const screenWithImage = (png: Buffer, except: Screen): Screen | undefined => {
|
||||
const sha = createHash("sha256").update(png).digest("hex");
|
||||
/**
|
||||
* The bytes a frame's screenshot is stored as, and their digest: an opened Duo's frame is turned upright once, however
|
||||
* often the frame is looked up or recorded. Keyed by the image, which every copy of the frame shares.
|
||||
*/
|
||||
const storedImages = new WeakMap<Buffer, { png: Buffer; sha256: string }>();
|
||||
const storedImage = (image: Buffer, landscape: boolean): { png: Buffer; sha256: string } => {
|
||||
let stored = storedImages.get(image);
|
||||
if (!stored) {
|
||||
const png = landscape ? turnedClockwise(image) : image;
|
||||
stored = { png, sha256: createHash("sha256").update(png).digest("hex") };
|
||||
storedImages.set(image, stored);
|
||||
}
|
||||
return stored;
|
||||
};
|
||||
|
||||
const screenWithImage = (sha: string, except: Screen): Screen | undefined => {
|
||||
const artifact = map.artifacts.find(item => item.kind === "screenshot" && item.sha256 === sha);
|
||||
if (!artifact) return undefined;
|
||||
const controls = structural(evidence.get(except.id)?.parts ?? new Set());
|
||||
@@ -491,7 +516,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
screen = { id: `scr_${key.slice(0, 16)}`, title: title ?? systemDialogTitle(frame.nodes) ?? `Screen ${map.screens.length + 1}`, start: map.screens.length === 0, terminal: false,
|
||||
depth, firstSeenAt: frame.capturedAt, lastSeenAt: frame.capturedAt, visits: 0, variantIds: [], observationIds: [], elementIds: [] };
|
||||
map.screens.push(screen); screensByKey.set(key, screen);
|
||||
} else if (title && /^Screen \d+$/.test(screen.title)) screen.title = title;
|
||||
} else if (title && isPlaceholderTitle(screen.title)) screen.title = title;
|
||||
screen.visits += 1; screen.lastSeenAt = frame.capturedAt;
|
||||
const seen = evidence.get(screen.id) ?? { parts: new Set<string>(), loadingOnly: true };
|
||||
for (const part of loose) seen.parts.add(part);
|
||||
@@ -529,11 +554,13 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
let screenshotArtifactId: string | undefined;
|
||||
if (withImage && frame.png) {
|
||||
// Hashed here rather than trusting the daemon's digest: the report validates artifacts against their bytes.
|
||||
const png = frame.png, sha = createHash("sha256").update(png).digest("hex");
|
||||
const { png, sha256: sha } = storedImage(frame.png, frame.landscape === true);
|
||||
const extent = frame.landscape ? frameExtent(frame.nodes) : undefined;
|
||||
let artifact = map.artifacts.find(item => item.kind === "screenshot" && item.sha256 === sha);
|
||||
if (!artifact) {
|
||||
artifact = { id: `art_${randomUUID().slice(0, 12)}`, kind: "screenshot" as const, path: `artifacts/screenshots/${sha}.png`, sha256: sha,
|
||||
bytes: png.length, mimeType: "image/png", ...(frame.width ? { width: frame.width } : {}), ...(frame.height ? { height: frame.height } : {}), createdAt: frame.capturedAt };
|
||||
bytes: png.length, mimeType: "image/png", ...(frame.width && frame.height ? frame.landscape ? { width: frame.height, height: frame.width } : { width: frame.width, height: frame.height } : {}),
|
||||
...(extent ? { turned: extent } : {}), createdAt: frame.capturedAt };
|
||||
map.artifacts.push(artifact); prepared.push({ artifact, bytes: png, destination: join(store.root, artifact.path) });
|
||||
}
|
||||
screenshotArtifactId = artifact.id;
|
||||
@@ -599,7 +626,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
// The same pixels as a screen already seen, with the same controls, is that screen: a tree read while Android was
|
||||
// still exposing a text node (AudioScape's playlist title missing from one read of two identical frames) is not new.
|
||||
if (created && latest.png) {
|
||||
const twin = screenWithImage(latest.png, screen);
|
||||
const twin = screenWithImage(storedImage(latest.png, latest.landscape === true).sha256, screen);
|
||||
if (twin) { mergeScreens(screen, twin); screen = twin; created = false; }
|
||||
}
|
||||
// Every screen gets an observation the moment it is seen, so elements and action attempts always point at a
|
||||
@@ -625,7 +652,7 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
|
||||
const actionFor = (input: Record<string, unknown>): { action: BlackBoxAction; kind: ActionKind; label: string; risk: ActionRisk } => {
|
||||
const action = String(input.action);
|
||||
const selector = input.identifier !== undefined ? { identifier: text(input.identifier, "identifier", 512) } : input.label !== undefined ? { label: text(input.label, "label", 512) } : undefined;
|
||||
const selector = input.identifier !== undefined ? { identifier: text(input.identifier, "identifier", TOOL_LIMITS.selector) } : input.label !== undefined ? { label: text(input.label, "label", TOOL_LIMITS.selector) } : undefined;
|
||||
const label = selector ? ("identifier" in selector ? selector.identifier : selector.label) : action;
|
||||
const risk = selector ? riskOf(label, allow) : "safe";
|
||||
switch (action) {
|
||||
@@ -637,13 +664,13 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
}
|
||||
if (!selector) throw failure("AGENT_ACTION_INVALID", "tap needs a label, an identifier, or frame coordinates.");
|
||||
return { action: { action: "tap", ...selector } as BlackBoxAction, kind: "press", label, risk };
|
||||
case "fill": if (!selector) throw failure("AGENT_ACTION_INVALID", "fill needs a label or identifier."); return { action: { action: "fill", text: text(input.text, "text", 2000), ...selector } as BlackBoxAction, kind: "fill", label, risk };
|
||||
case "type": return { action: { action: "type", text: text(input.text, "text", 2000) }, kind: "fill", label: "type", risk: "safe" };
|
||||
case "fill": if (!selector) throw failure("AGENT_ACTION_INVALID", "fill needs a label or identifier."); return { action: { action: "fill", text: text(input.text, "text", TOOL_LIMITS.text), ...selector } as BlackBoxAction, kind: "fill", label, risk };
|
||||
case "type": return { action: { action: "type", text: text(input.text, "text", TOOL_LIMITS.text) }, kind: "fill", label: "type", risk: "safe" };
|
||||
case "scroll": return { action: { action: "scroll", direction: input.direction as "up" }, kind: "swipe", label: `scroll ${String(input.direction)}`, risk: "safe" };
|
||||
case "wait": return { action: { action: "wait", milliseconds: Number(input.milliseconds) || 1000 }, kind: "other", label: "wait", risk: "safe" };
|
||||
case "back": return { action: { action: "back" }, kind: "back", label: "back", risk: "safe" };
|
||||
case "dismiss-keyboard": return { action: { action: "dismiss-keyboard" }, kind: "dismiss", label: "dismiss keyboard", risk: "safe" };
|
||||
case "open": return { action: { action: "open", url: text(input.url, "url", 2000) }, kind: "deep-link", label: text(input.url, "url", 2000), risk: "unknown" };
|
||||
case "open": return { action: { action: "open", url: text(input.url, "url", TOOL_LIMITS.url) }, kind: "deep-link", label: text(input.url, "url", TOOL_LIMITS.url), risk: "unknown" };
|
||||
default: throw failure("AGENT_ACTION_INVALID", `Unknown action ${action}.`);
|
||||
}
|
||||
};
|
||||
@@ -707,8 +734,12 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
const role = roleOf(under);
|
||||
const label = str(under.label) ?? "";
|
||||
const what = role === "field" ? "field" : role === "button" || role === "tab" || role === "link" ? (cleanLabel(label) ? role : `icon ${role}`) : /\p{L}/u.test(cleanLabel(label)) ? "item" : "icon";
|
||||
const row = point.y < 1 / 3 ? "top" : point.y < 2 / 3 ? "" : "bottom";
|
||||
const column = point.x < 1 / 3 ? "left" : point.x < 2 / 3 ? "" : "right";
|
||||
// The place is read off the screen's stored screenshot, which an opened Duo's frames are turned upright for: there the
|
||||
// portrait point (x, y) shows at (1 - y, x). A frame read without an image says nothing, so the screen's screenshot is asked.
|
||||
const turned = latest!.landscape || map.observations.some(item => item.screenId === currentScreenId && map.artifacts.find(artifact => artifact.id === item.screenshotArtifactId)?.turned);
|
||||
const shown = turned ? { x: 1 - point.y, y: point.x } : point;
|
||||
const row = shown.y < 1 / 3 ? "top" : shown.y < 2 / 3 ? "" : "bottom";
|
||||
const column = shown.x < 1 / 3 ? "left" : shown.x < 2 / 3 ? "" : "right";
|
||||
const where = [row, column].filter(Boolean).join(" ") || "centre";
|
||||
return `${what}, ${where}`;
|
||||
};
|
||||
@@ -749,10 +780,10 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
const frameId = text(input.frameId, "frameId", 256);
|
||||
if (!latest?.png && latest?.imageUnavailable?.code === "CAPTURE_SENSITIVE_INPUT") throw failure("CAPTURE_SENSITIVE_INPUT", latest.imageUnavailable.message, true);
|
||||
if (!latest?.png || latest.frameId !== frameId) throw failure("AGENT_FRAME_STALE", "Observe with an image first; capture must name the latest frame.", true);
|
||||
const title = text(input.title, "title", 80).trim();
|
||||
const title = text(input.title, "title", TOOL_LIMITS.title).trim();
|
||||
if (captures >= options.maxScreens) { stop("budget-screens"); abort.abort(failure("EXPLORATION_BUDGET_EXHAUSTED", "Screen budget exhausted.")); throw abort.signal.reason; }
|
||||
const { screen } = locate(latest, title);
|
||||
const state = str(input.state, 80);
|
||||
const state = str(input.state, TOOL_LIMITS.state);
|
||||
if (!state && map.observations.some(item => item.screenId === screen.id && item.screenshotArtifactId)) {
|
||||
if (title) screen.title = title;
|
||||
return toolResult({ ok: true, duplicate: true, screenId: screen.id, title: screen.title, message: "This screen is already captured (its title is updated); pass state to record a variant." });
|
||||
@@ -832,9 +863,9 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
await persist();
|
||||
return toolResult({ ok: !error, outcome, ...(error ? { error } : {}), ...view, ...progress().budget });
|
||||
}
|
||||
if (name === "finish") { stop("finished", text(input.summary, "summary", 4000)); abort.abort(new Error("finished")); return toolResult({ ok: true }); }
|
||||
if (name === "finish") { stop("finished", text(input.summary, "summary", TOOL_LIMITS.summary)); abort.abort(new Error("finished")); return toolResult({ ok: true }); }
|
||||
if (name === "stop_blocked") {
|
||||
const reason = text(input.reason, "reason", 1000);
|
||||
const reason = text(input.reason, "reason", TOOL_LIMITS.reason);
|
||||
map.gaps.push({ id: `gap_${randomUUID().slice(0, 8)}`, kind: "coverage", status: "open", message: reason, ...(currentScreenId ? { screenId: currentScreenId } : {}), createdAt: timestamp() });
|
||||
await persist();
|
||||
stop("blocked", reason); abort.abort(new Error("blocked")); return toolResult({ ok: true });
|
||||
@@ -856,8 +887,9 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
if (!result) stop(budget ? "budget-time" : "model-error", undefined, budget ? undefined : fatal ?? error);
|
||||
}
|
||||
if (result!.stopReason !== "finished" && result!.stopReason !== "blocked") {
|
||||
const note = result!.stopReason === "model-error" ? undefined : options.budgetNote?.();
|
||||
map.gaps.push({ id: `gap_${randomUUID().slice(0, 8)}`, kind: result!.stopReason === "model-error" ? "other" : "coverage", status: "open",
|
||||
message: result!.stopReason === "model-error" ? `The exploration stopped on an error: ${result!.error instanceof Error ? result!.error.message.slice(0, 300) : "unknown"}` : `The exploration stopped at its ${result!.stopReason.replace("budget-", "")} budget with ${map.frontier.filter(item => item.status === "pending").length} controls unexplored.`,
|
||||
message: result!.stopReason === "model-error" ? `The exploration stopped on an error: ${result!.error instanceof Error ? result!.error.message.slice(0, 300) : "unknown"}` : `The exploration stopped at its ${result!.stopReason.replace("budget-", "")} budget with ${map.frontier.filter(item => item.status === "pending").length} controls unexplored.${note ? ` ${note}` : ""}`,
|
||||
createdAt: timestamp() });
|
||||
}
|
||||
for (const item of map.frontier) if (item.status === "claimed") item.status = "pending";
|
||||
@@ -870,10 +902,10 @@ export async function runBlackBoxExploration(options: BlackBoxExplorerOptions):
|
||||
function nameUncapturedScreens(map: ExploreMap): void {
|
||||
const titles = new Map(map.screens.map(screen => [screen.id, screen.title]));
|
||||
for (const screen of map.screens) {
|
||||
if (!/^Screen \d+$/.test(screen.title)) continue;
|
||||
if (!isPlaceholderTitle(screen.title)) continue;
|
||||
const arrival = map.actionAttempts.find(attempt => attempt.targetScreenId === screen.id && attempt.outcome === "transition" && attempt.sourceScreenId !== screen.id);
|
||||
const from = arrival ? titles.get(arrival.sourceScreenId) : undefined;
|
||||
if (!arrival || !from || /^Screen \d+$/.test(from)) continue;
|
||||
if (!arrival || !from || isPlaceholderTitle(from)) continue;
|
||||
const step = arrival.action.input ? "typing" : arrival.action.label;
|
||||
screen.title = `${from} → ${step}`.slice(0, 80);
|
||||
titles.set(screen.id, screen.title);
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { CiError } from "../shared/ci-errors.js";
|
||||
import type { CiAgentSessionRunRequest, CiAgentSessionRunResult } from "../capture/agent/session.js";
|
||||
import { SETUP_SELECTOR_MAX, isSetupInput, isSetupSelector } from "../capture/setup-actions.js";
|
||||
import type { CaptureSetupAction } from "../config/explore-config.js";
|
||||
import { explorationSecretNames } from "../config/exploration-inputs.js";
|
||||
import { TOOL_LIMITS, isPlaceholderTitle } from "./explorer.js";
|
||||
|
||||
export type ReplayPlatform = "ios" | "android";
|
||||
type Selector = { label: string; identifier?: never } | { identifier: string; label?: never };
|
||||
type ReplayAction =
|
||||
| ({ action: "tap" } & Selector)
|
||||
| ({ action: "fill"; text: string } & Selector)
|
||||
| { action: "type"; text: string }
|
||||
| { action: "scroll"; direction: "up" | "down" | "left" | "right" | "top" | "bottom" }
|
||||
| { action: "wait"; milliseconds: number }
|
||||
| { action: "open"; url: string }
|
||||
| { action: "back" }
|
||||
| { action: "dismiss-keyboard" };
|
||||
type ReplayCapture = { capture: string; state?: string; description?: string };
|
||||
/** `platforms` limits a step to those platforms; an `optional` action that fails is skipped instead of ending the replay. */
|
||||
export type ReplayStep = (ReplayCapture & { platforms?: ReplayPlatform[] }) | (ReplayAction & { platforms?: ReplayPlatform[]; optional?: boolean });
|
||||
export type ReplayPlan = { steps: ReplayStep[] };
|
||||
|
||||
export type ReplaySession = {
|
||||
readonly provider: "replay";
|
||||
readonly model: string;
|
||||
preflight(): Promise<void>;
|
||||
run(request: CiAgentSessionRunRequest): Promise<CiAgentSessionRunResult>;
|
||||
/** After a budget stop, the planned screens the run did not reach, for the explorer's gap. */
|
||||
budgetNote(): string | undefined;
|
||||
};
|
||||
|
||||
const DIRECTIONS = new Set(["up", "down", "left", "right", "top", "bottom"]);
|
||||
/**
|
||||
* The fields each kind of step takes. Anything else is refused: a typo (`platform`, `optinal`) would otherwise do nothing,
|
||||
* and a field `act` reads another way (a coordinate tap's `frameId`, `x`, `y` and `target`) would change what is tapped.
|
||||
*/
|
||||
const FIELDS = new Map<string, readonly string[]>([
|
||||
["capture", ["capture", "state", "description", "platforms"]],
|
||||
["tap", ["action", "label", "identifier", "platforms", "optional"]],
|
||||
["fill", ["action", "label", "identifier", "text", "platforms", "optional"]],
|
||||
["type", ["action", "text", "platforms", "optional"]],
|
||||
["scroll", ["action", "direction", "platforms", "optional"]],
|
||||
["wait", ["action", "milliseconds", "platforms", "optional"]],
|
||||
["open", ["action", "url", "platforms", "optional"]],
|
||||
["back", ["action", "platforms", "optional"]],
|
||||
["dismiss-keyboard", ["action", "platforms", "optional"]],
|
||||
]);
|
||||
const ACTIONS = [...FIELDS.keys()].filter(kind => kind !== "capture");
|
||||
|
||||
function invalid(message: string): CiError {
|
||||
return new CiError({ code: "REPLAY_PLAN_INVALID", stage: "exploration", message, retryable: false,
|
||||
nextAction: "Fix the replay plan: a JSON object with a steps array of actions ({\"action\":\"tap\",\"label\":\"Settings\"}) and captures ({\"capture\":\"Settings\"})." });
|
||||
}
|
||||
|
||||
const nonEmpty = (value: unknown, max: number): value is string => typeof value === "string" && value.trim().length > 0 && value.length <= max;
|
||||
|
||||
function parseStep(step: unknown, at: string): ReplayStep {
|
||||
if (!step || typeof step !== "object" || Array.isArray(step)) throw invalid(`${at} is not an object.`);
|
||||
const entry = step as Record<string, unknown>;
|
||||
const kind = "capture" in entry ? "capture" : typeof entry.action === "string" && FIELDS.has(entry.action) ? entry.action : undefined;
|
||||
if (!kind) throw invalid(`${at} needs a capture title or an action (${ACTIONS.join(", ")}).`);
|
||||
const fields = FIELDS.get(kind)!;
|
||||
const unknown = Object.keys(entry).find(key => !fields.includes(key));
|
||||
if (unknown !== undefined) throw invalid(`${at} (${kind}) has a field ${JSON.stringify(unknown.slice(0, 40))} it does not take; a ${kind} step takes ${fields.join(", ")}.`);
|
||||
if (entry.platforms !== undefined && (!Array.isArray(entry.platforms) || !entry.platforms.length || entry.platforms.some(item => item !== "ios" && item !== "android"))) {
|
||||
throw invalid(`${at}: platforms must list ios and/or android.`);
|
||||
}
|
||||
if (entry.optional !== undefined && typeof entry.optional !== "boolean") throw invalid(`${at}: optional must be true or false.`);
|
||||
const extra = { ...(entry.platforms ? { platforms: [...entry.platforms] as ReplayPlatform[] } : {}), ...(entry.optional !== undefined ? { optional: entry.optional as boolean } : {}) };
|
||||
if (kind === "capture") {
|
||||
if (!nonEmpty(entry.capture, TOOL_LIMITS.title)) throw invalid(`${at}: capture must be a screen title of at most ${TOOL_LIMITS.title} characters.`);
|
||||
if (entry.state !== undefined && !nonEmpty(entry.state, TOOL_LIMITS.state)) throw invalid(`${at}: state must be a string of at most ${TOOL_LIMITS.state} characters.`);
|
||||
if (entry.description !== undefined && !nonEmpty(entry.description, TOOL_LIMITS.description)) throw invalid(`${at}: description must be a string of at most ${TOOL_LIMITS.description} characters.`);
|
||||
return { capture: entry.capture.trim(), ...(entry.state !== undefined ? { state: entry.state as string } : {}),
|
||||
...(entry.description !== undefined ? { description: entry.description as string } : {}), ...extra };
|
||||
}
|
||||
// The device selects by the setup executor's rules, which are stricter than what `act` itself accepts.
|
||||
const selector = (): Selector => {
|
||||
const keys = (["label", "identifier"] as const).filter(key => entry[key] !== undefined);
|
||||
if (keys.length !== 1) throw invalid(`${at}: ${kind} needs exactly one of label or identifier.`);
|
||||
const key = keys[0]!, value = entry[key];
|
||||
if (!isSetupSelector(value) || value.length > TOOL_LIMITS.selector) throw invalid(`${at}: ${key} must be 1–${SETUP_SELECTOR_MAX} characters without control characters.`);
|
||||
return key === "label" ? { label: value } : { identifier: value };
|
||||
};
|
||||
const input = (): string => {
|
||||
if (!nonEmpty(entry.text, TOOL_LIMITS.text) || !isSetupInput(entry.text)) throw invalid(`${at}: ${kind} needs text: one line of at most ${TOOL_LIMITS.text} characters.`);
|
||||
try { explorationSecretNames(entry.text); }
|
||||
catch (error) { throw invalid(`${at}: ${error instanceof Error ? error.message : String(error)} Write each input as {{secret:NAME}} with an uppercase environment variable name that is not a provider, Expo, or CI credential.`); }
|
||||
return entry.text;
|
||||
};
|
||||
switch (kind) {
|
||||
case "tap": return { action: "tap", ...selector(), ...extra };
|
||||
case "fill": return { action: "fill", ...selector(), text: input(), ...extra };
|
||||
case "type": return { action: "type", text: input(), ...extra };
|
||||
case "scroll":
|
||||
if (typeof entry.direction !== "string" || !DIRECTIONS.has(entry.direction)) throw invalid(`${at}: scroll needs a direction (${[...DIRECTIONS].join(", ")}).`);
|
||||
return { action: "scroll", direction: entry.direction as "up", ...extra };
|
||||
case "wait":
|
||||
if (!(Number.isInteger(entry.milliseconds) && Number(entry.milliseconds) >= 1 && Number(entry.milliseconds) <= TOOL_LIMITS.waitMs)) {
|
||||
throw invalid(`${at}: wait needs milliseconds between 1 and ${TOOL_LIMITS.waitMs}.`);
|
||||
}
|
||||
return { action: "wait", milliseconds: entry.milliseconds as number, ...extra };
|
||||
case "open":
|
||||
if (!nonEmpty(entry.url, TOOL_LIMITS.url)) throw invalid(`${at}: open needs a url of at most ${TOOL_LIMITS.url} characters.`);
|
||||
return { action: "open", url: entry.url, ...extra };
|
||||
default: return { action: kind as "back" | "dismiss-keyboard", ...extra };
|
||||
}
|
||||
}
|
||||
|
||||
/** Checks a plan and keeps only the fields each step takes, so what reaches `act` is what the plan documents. */
|
||||
export function parseReplayPlan(value: unknown): ReplayPlan {
|
||||
if (!value || typeof value !== "object" || !Array.isArray((value as { steps?: unknown }).steps)) throw invalid("The replay plan has no steps array.");
|
||||
const unknown = Object.keys(value).find(key => key !== "steps");
|
||||
if (unknown !== undefined) throw invalid(`The replay plan has a field ${JSON.stringify(unknown.slice(0, 40))} it does not take; it takes only steps.`);
|
||||
const steps = (value as { steps: unknown[] }).steps;
|
||||
if (!steps.length) throw invalid("The replay plan has no steps.");
|
||||
return { steps: steps.map((step, index) => parseStep(step, `Step ${index + 1}`)) };
|
||||
}
|
||||
|
||||
export async function readReplayPlan(path: string): Promise<ReplayPlan> {
|
||||
let text: string;
|
||||
try { text = await readFile(path, "utf8"); }
|
||||
catch (error) { throw invalid(`The replay plan ${path} could not be read: ${error instanceof Error ? error.message : String(error)}`); }
|
||||
let value: unknown;
|
||||
try { value = JSON.parse(text); }
|
||||
catch (error) { throw invalid(`The replay plan ${path} is not JSON: ${error instanceof Error ? error.message : String(error)}`); }
|
||||
return parseReplayPlan(value);
|
||||
}
|
||||
|
||||
/** The plan's fill and type steps, whose {{secret:NAME}} inputs must be set before any device starts, as the hints' are. */
|
||||
export function replaySetup(plan: ReplayPlan): CaptureSetupAction[] {
|
||||
return plan.steps.flatMap(step => "action" in step && (step.action === "fill" || step.action === "type") ? [step as CaptureSetupAction] : []);
|
||||
}
|
||||
|
||||
const stepsFor = (plan: ReplayPlan, platform: ReplayPlatform) => plan.steps.map((step, index) => ({ step, number: index + 1 }))
|
||||
.filter(({ step }) => !step.platforms || step.platforms.includes(platform));
|
||||
|
||||
/**
|
||||
* Refuses, before any device starts, a plan with nothing to do on this platform or one the budgets cannot hold: a run that
|
||||
* would stop at its budget partway through the plan is better raised or shortened up front.
|
||||
*/
|
||||
export function assertReplayFits(plan: ReplayPlan, platform: ReplayPlatform, limits: { maxScreens: number; maxActions: number; maxTurns: number }): void {
|
||||
const steps = stepsFor(plan, platform);
|
||||
if (!steps.length) {
|
||||
throw new CiError({ code: "REPLAY_PLAN_EMPTY", stage: "exploration", message: `No step of the replay plan runs on ${platform}: each one is limited to ${platform === "ios" ? "android" : "ios"} by its platforms.`,
|
||||
nextAction: `Add steps for ${platform} to the plan, or run it on the platform it was written for.` });
|
||||
}
|
||||
const captures = steps.filter(({ step }) => "capture" in step).length;
|
||||
const actions = steps.filter(({ step }) => "action" in step && step.action !== "wait").length;
|
||||
// A screen counts against the budget when it is first photographed, on arrival, and a capture is refused once the budget
|
||||
// is used up, so the last capture needs one screen to spare.
|
||||
const over = [
|
||||
...(captures >= limits.maxScreens ? [{ text: `${captures} captures, which need a screen budget of at least ${captures + 1} (it is ${limits.maxScreens})`, flag: `--max-screens ${captures + 1}` }] : []),
|
||||
...(actions > limits.maxActions ? [{ text: `${actions} actions, more than the action budget of ${limits.maxActions}`, flag: `--max-actions ${actions}` }] : []),
|
||||
];
|
||||
if (over.length) {
|
||||
throw new CiError({ code: "REPLAY_PLAN_OVER_BUDGET", stage: "exploration", message: `On ${platform} the replay plan takes ${over.map(item => item.text).join(" and ")}.`,
|
||||
nextAction: `Pass ${over.map(item => item.flag).join(" ")} (or more: each screen the plan reaches without capturing it also counts), or shorten the plan.` });
|
||||
}
|
||||
// Each capture is an observe and a capture call, each action one act call, and the run ends with get_progress and finish.
|
||||
const turns = steps.length + captures + 2;
|
||||
if (turns > limits.maxTurns) {
|
||||
throw new CiError({ code: "REPLAY_PLAN_OVER_BUDGET", stage: "exploration", message: `On ${platform} the replay plan takes ${turns} tool calls, more than the ${limits.maxTurns} that the screen and action budgets allow.`,
|
||||
nextAction: "Raise --max-actions, or use fewer wait steps." });
|
||||
}
|
||||
}
|
||||
|
||||
function describeAction(step: ReplayAction): string {
|
||||
if ("label" in step && step.label) return `${step.action} "${step.label}"`;
|
||||
if ("identifier" in step && step.identifier) return `${step.action} #${step.identifier}`;
|
||||
if (step.action === "scroll") return `scroll ${step.direction}`;
|
||||
return step.action;
|
||||
}
|
||||
|
||||
/** Only the fields `act` documents for that action, whatever else the step object carries. */
|
||||
function actInput(step: ReplayAction): Record<string, unknown> {
|
||||
const selector = "label" in step && step.label !== undefined ? { label: step.label } : "identifier" in step && step.identifier !== undefined ? { identifier: step.identifier } : {};
|
||||
switch (step.action) {
|
||||
case "tap": return { action: "tap", ...selector };
|
||||
case "fill": return { action: "fill", ...selector, text: step.text };
|
||||
case "type": return { action: "type", text: step.text };
|
||||
case "scroll": return { action: "scroll", direction: step.direction };
|
||||
case "wait": return { action: "wait", milliseconds: step.milliseconds };
|
||||
case "open": return { action: "open", url: step.url };
|
||||
default: return { action: step.action };
|
||||
}
|
||||
}
|
||||
|
||||
/** As many names as fit in `room` characters, then "and N more"; a count of them when not even the first fits. */
|
||||
function listed(names: readonly string[], room: number, noun: string): string {
|
||||
let text = "";
|
||||
for (const [index, name] of names.entries()) {
|
||||
const next = text ? `${text}, ${name}` : name;
|
||||
const left = names.length - index - 1;
|
||||
if (next.length + (left ? ` and ${left} more`.length : 0) > room) return text ? `${text} and ${names.length - index} more` : `${names.length} ${noun}`;
|
||||
text = next;
|
||||
}
|
||||
return text;
|
||||
}
|
||||
|
||||
const sameTitle = (left: string, right: string): boolean => left.toLowerCase() === right.toLowerCase();
|
||||
|
||||
/**
|
||||
* Runs a fixed plan through the explorer's own tools in place of a model: the same observe, act, and capture calls,
|
||||
* so the map and report are those of a model run, but every run takes the same steps. A step that fails does not
|
||||
* fail the run: the report keeps what was captured and names the screens that were not.
|
||||
*/
|
||||
export function createReplaySession(plan: ReplayPlan, platform: ReplayPlatform): ReplaySession {
|
||||
const steps = stepsFor(plan, platform);
|
||||
const title = (step: ReplayCapture) => step.state ? `${step.capture} (${step.state})` : step.capture;
|
||||
const missed: string[] = [];
|
||||
let position = 0;
|
||||
/** What the plan meant to capture and has not: the screens still ahead first, then the ones earlier steps missed. */
|
||||
const notCaptured = () => [...steps.slice(position).flatMap(({ step }) => "capture" in step ? [title(step)] : []), ...missed];
|
||||
return {
|
||||
provider: "replay",
|
||||
model: "replay plan",
|
||||
async preflight() {},
|
||||
budgetNote() {
|
||||
const names = notCaptured();
|
||||
return names.length ? `Not captured: ${listed(names, TOOL_LIMITS.reason, "planned screens")}.` : undefined;
|
||||
},
|
||||
async run(request) {
|
||||
const signal = request.signal ?? new AbortController().signal;
|
||||
const call = async (name: string, input: Record<string, unknown>) => {
|
||||
const result = await request.callTool(name, input, signal);
|
||||
const text = result.content.find(block => block.type === "text");
|
||||
const value = text && "text" in text ? JSON.parse(text.text) as Record<string, unknown> : {};
|
||||
return { failed: Boolean(result.isError) || value.ok === false, value };
|
||||
};
|
||||
const why = (value: Record<string, unknown>) => String(value.error ?? value.reason ?? value.message ?? "it failed");
|
||||
// A refused finish or stop_blocked would otherwise end the session with no result, which the explorer reports as a turn budget stop.
|
||||
const end = async (name: "finish" | "stop_blocked", input: Record<string, unknown>) => {
|
||||
const ended = await call(name, input);
|
||||
if (!ended.failed) return;
|
||||
if (name === "finish") return end("stop_blocked", { reason: `The replay could not finish (${why(ended.value)}). ${String(input.summary)}`.slice(0, TOOL_LIMITS.reason) });
|
||||
throw new CiError({ code: "REPLAY_STOP_REFUSED", stage: "exploration", message: `The explorer refused the replay's ${name}: ${why(ended.value)}`.slice(0, 600),
|
||||
nextAction: "Report this with the replay plan; the screens captured so far are in the map." });
|
||||
};
|
||||
/** Screen id to the title the plan captured it under. */
|
||||
const planned = new Map<string, string>();
|
||||
const skipped: string[] = [];
|
||||
missed.length = 0;
|
||||
try {
|
||||
for (const [index, { step, number }] of steps.entries()) {
|
||||
position = index;
|
||||
if ("capture" in step) {
|
||||
const seen = await call("observe", { includeImage: true });
|
||||
const frameId = typeof seen.value.frameId === "string" ? seen.value.frameId : undefined;
|
||||
const known = typeof seen.value.screenId === "string" && !step.state ? planned.get(seen.value.screenId) : undefined;
|
||||
// The plan already captured this screen: a second capture would only rename it, so it is not made.
|
||||
if (known !== undefined) {
|
||||
if (!sameTitle(known, step.capture)) missed.push(`${title(step)} (looked the same as ${known})`);
|
||||
continue;
|
||||
}
|
||||
const captured = frameId && !seen.failed ? await call("capture", { frameId, title: step.capture,
|
||||
...(step.state ? { state: step.state } : {}), ...(step.description ? { description: step.description } : {}) }) : undefined;
|
||||
const screenId = typeof captured?.value.screenId === "string" ? captured.value.screenId : undefined;
|
||||
if (!captured || captured.failed || !screenId) missed.push(title(step));
|
||||
else if (!planned.has(screenId)) planned.set(screenId, step.capture);
|
||||
continue;
|
||||
}
|
||||
const acted = await call("act", actInput(step));
|
||||
if (!acted.failed) continue;
|
||||
if (step.optional) { skipped.push(`step ${number} (${describeAction(step)})`); continue; }
|
||||
const head = `Replay step ${number} (${describeAction(step)}) failed: ${why(acted.value)}`.slice(0, 600);
|
||||
const names = notCaptured();
|
||||
const lead = ". Not captured: ";
|
||||
await end("stop_blocked", { reason: head + (names.length ? `${lead}${listed(names, TOOL_LIMITS.reason - head.length - lead.length, "planned screens")}` : "") });
|
||||
return { inputTokens: 0, outputTokens: 0 };
|
||||
}
|
||||
position = steps.length;
|
||||
// The map as it stands: screens merged into others are gone, and each keeps its latest title. One reached but never
|
||||
// captured is still "Screen N" here and is named after the path to it only once the run ends, so it is counted.
|
||||
const progress = await call("get_progress", {});
|
||||
const screens = Array.isArray(progress.value.captured) ? progress.value.captured as Array<{ id?: unknown; title?: unknown }> : [];
|
||||
const unplanned = screens.filter(screen => typeof screen.id === "string" && !planned.has(screen.id)).map(screen => typeof screen.title === "string" ? screen.title : "");
|
||||
const named = unplanned.filter(name => name && !isPlaceholderTitle(name));
|
||||
const unnamed = unplanned.length - named.length;
|
||||
const reached = [...named, ...(unnamed ? [`${unnamed} unnamed screen${unnamed === 1 ? "" : "s"}`] : [])];
|
||||
const sections = ([["Not captured", missed, "planned screens"], ["Optional steps skipped because they failed", skipped, "steps"],
|
||||
["Also reached, not in the plan", reached, "screens"]] as const).filter(([, names]) => names.length);
|
||||
let summary = `Replayed ${steps.length} planned steps on ${platform}.${missed.length ? "" : " Every planned screen was captured."}`;
|
||||
for (const [index, [label, names, noun]] of sections.entries()) {
|
||||
// Each later section keeps room for its label and a count, so a long list of misses does not crowd out the rest.
|
||||
const reserve = sections.slice(index + 1).reduce((sum, [later]) => sum + later.length + 32, 0);
|
||||
const lead = ` ${label}: `;
|
||||
summary += `${lead}${listed(names, TOOL_LIMITS.summary - summary.length - lead.length - 1 - reserve, noun)}.`;
|
||||
}
|
||||
await end("finish", { summary });
|
||||
} catch (error) {
|
||||
if (signal.aborted) return { inputTokens: 0, outputTokens: 0 };
|
||||
throw error;
|
||||
}
|
||||
return { inputTokens: 0, outputTokens: 0 };
|
||||
},
|
||||
};
|
||||
}
|
||||
+83
-10
@@ -5,6 +5,7 @@ import { join } from "node:path";
|
||||
import { createCiAgentSession, type CiAgentSession, type CiAgentSessionConfig } from "../capture/agent/session.js";
|
||||
import { createLocalCiAgentSession } from "../capture/agent/session-local.js";
|
||||
import type { CiAgentProviderName } from "../capture/agent/provider.js";
|
||||
import { assertReplayFits, createReplaySession, parseReplayPlan, replaySetup, type ReplayPlan } from "./replay.js";
|
||||
import { CONTROLLER_SESSION, baseEnvironment, isRecord } from "../capture/common.js";
|
||||
import { controllerData, defaultRunner, openedSimulatorDevice } from "../capture/controller.js";
|
||||
import type { CommandRequest, CommandResult } from "../capture/run.js";
|
||||
@@ -16,8 +17,10 @@ import { agentDeviceLauncher, ensureArgentToolServer, ensureController } from ".
|
||||
import { agentDeviceClaims, approveLinkScheme, createLocalSimulator, deleteLocalEmulator, deleteLocalSimulator, hostEmulatorAbi, installOnLocalEmulator, installOnLocalSimulator, startLocalArgentServer,
|
||||
startLocalEmulator, stopLocalAgentDevice, stopLocalArgentServer, waitForLocalEmulator, waitForLocalSimulator, type LocalArgentServer } from "../capture/local-simulator.js";
|
||||
import { openLocalCaptureRecord, sweepLocalCaptures } from "../capture/local-registry.js";
|
||||
import { screenSpace } from "../capture/route-capture.js";
|
||||
import { approveSessionUrlSchemes, simulatorControlActions, type EasCommand } from "../capture/simulator-control.js";
|
||||
import { DEFAULT_ANDROID_DEVICE } from "../config/explore-config.js";
|
||||
import { DUO_PANELS } from "../contracts/capture.js";
|
||||
import { createExplorationInputs } from "../config/exploration-inputs.js";
|
||||
import { ExploreStore, getCoverage } from "../map/store.js";
|
||||
import { whoamiAccounts } from "../report/preview-deploy.js";
|
||||
@@ -57,6 +60,8 @@ export type BinaryExploreOptions = {
|
||||
simulator?: "eas" | "local";
|
||||
environment?: NodeJS.ProcessEnv;
|
||||
runner?: typeof defaultRunner;
|
||||
/** Fixed steps to take instead of a model's; no model access is needed. */
|
||||
replay?: ReplayPlan;
|
||||
createSession?: (config: CiAgentSessionConfig, options: { env: NodeJS.ProcessEnv; auth: ModelAuth }) => CiAgentSession;
|
||||
progress?: (message: string) => void;
|
||||
now?: () => number;
|
||||
@@ -68,7 +73,7 @@ export type BinaryExploreResult = {
|
||||
archive: Omit<InspectedArchive, "path"> & { path: string };
|
||||
session: { id: string; platform: Platform; device: string; owner: string; project: string };
|
||||
runtime: ActiveSimulatorRuntime;
|
||||
model: { provider: CiAgentProviderName; model: string; auth: "local-login" | "api-key" };
|
||||
model: { provider: CiAgentProviderName | "replay"; model: string; auth: "local-login" | "api-key" | "none" };
|
||||
exploration: BlackBoxExplorerResult;
|
||||
durationMs: number;
|
||||
};
|
||||
@@ -162,8 +167,8 @@ export function resolveSessionApp(runner: typeof defaultRunner, connection: Simu
|
||||
* App-test inputs: each {{secret:NAME}} in the hints names an environment variable whose value Explore types for the
|
||||
* model. Read before any session starts, so a missing one costs nothing.
|
||||
*/
|
||||
export function hintInputs(hints: string | undefined, environment: NodeJS.ProcessEnv): ReturnType<typeof createExplorationInputs> {
|
||||
try { return createExplorationInputs({ instructions: hints ?? "", environment }); }
|
||||
export function hintInputs(hints: string | undefined, environment: NodeJS.ProcessEnv, replay?: ReplayPlan): ReturnType<typeof createExplorationInputs> {
|
||||
try { return createExplorationInputs({ instructions: hints ?? "", ...(replay ? { setup: replaySetup(replay) } : {}), environment }); }
|
||||
catch (error) {
|
||||
if (!(error instanceof CiError) || !error.code.startsWith("EXPLORATION_SECRET_")) throw error;
|
||||
const name = typeof error.details?.secretName === "string" ? error.details.secretName : "NAME";
|
||||
@@ -194,6 +199,36 @@ function modelSession(provider: CiAgentProviderName, auth: ModelAuth, options: M
|
||||
return { session, provider, model: config.model, auth };
|
||||
}
|
||||
|
||||
/** The turns the model gets: three for each action and two for each screen, plus a few to look around and finish. */
|
||||
const turnBudget = (limits: { maxScreens: number; maxActions: number }): number => limits.maxActions * 3 + limits.maxScreens * 2 + 20;
|
||||
|
||||
/**
|
||||
* A replay plan is checked here as well as by the CLI, for callers that pass one directly. It takes the model's place, so
|
||||
* the options that only steer a model would do nothing; the plan's own fill and type steps name the secrets it types.
|
||||
*/
|
||||
function replayPlan(options: Pick<BinaryExploreOptions, "replay" | "provider" | "model" | "hints">): ReplayPlan | undefined {
|
||||
if (!options.replay) return undefined;
|
||||
const ignored = (["provider", "model", "hints"] as const).filter(name => options[name] !== undefined);
|
||||
if (ignored.length) {
|
||||
throw binaryError("REPLAY_OPTIONS_IGNORED", `--replay takes the model's place, so ${ignored.map(name => `--${name}`).join(" and ")} would do nothing.`,
|
||||
`Drop ${ignored.map(name => `--${name}`).join(" and ")}. A {{secret:NAME}} in the plan's fill and type steps is read from the environment, as one in --hints is.`);
|
||||
}
|
||||
const plan = parseReplayPlan(options.replay);
|
||||
for (const [index, step] of plan.steps.entries()) {
|
||||
if (!("action" in step) || step.action !== "open") continue;
|
||||
try { assertAppUrl(step.url); }
|
||||
catch (error) { throw new CiError({ code: "REPLAY_PLAN_INVALID", stage: "exploration", message: `Step ${index + 1}: ${error instanceof Error ? error.message : String(error)}`, nextAction: "Open only the app's own URLs in the replay plan." }); }
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
|
||||
/** A replay plan takes the model's place: no model access is checked or used. */
|
||||
function replayModel(plan: ReplayPlan, platform: Platform, limits: { maxScreens: number; maxActions: number }) {
|
||||
assertReplayFits(plan, platform, { ...limits, maxTurns: turnBudget(limits) });
|
||||
const session = createReplaySession(plan, platform);
|
||||
return { session, budgetNote: session.budgetNote, provider: "replay" as const, model: "replay plan", auth: "none" as const };
|
||||
}
|
||||
|
||||
/** Local `claude`/`codex` login unless an API key is present; preflight proves access before a simulator is paid for. */
|
||||
export function selectModelSession(options: ModelSessionOptions, environment: NodeJS.ProcessEnv) {
|
||||
const provider: CiAgentProviderName = options.provider ?? (environment.OPENAI_API_KEY?.trim() && !environment.ANTHROPIC_API_KEY?.trim() ? "openai" : "anthropic");
|
||||
@@ -260,7 +295,7 @@ export type ConnectedSessionExploration = {
|
||||
limits: { maxDurationMinutes: number; maxScreens: number; maxActions: number };
|
||||
store: ExploreStore;
|
||||
map: ExploreMap;
|
||||
model: { session: CiAgentSession };
|
||||
model: { session: Pick<CiAgentSession, "run">; budgetNote?: () => string | undefined };
|
||||
/** `eas simulator:get --json` for the attached session; descriptive (OS version) only. */
|
||||
sessionJson: () => string;
|
||||
/** agent-device session name on the daemon; explore:session reuses the caller's session so the device is not "in use" by another. */
|
||||
@@ -374,6 +409,20 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
|
||||
catch { return undefined; }
|
||||
};
|
||||
const foregroundElsewhere = (): boolean => { const state = foregroundState(); return state ? leftForeground(state, bundleId) : false; };
|
||||
let unmatchedPanel = false;
|
||||
/** Whether the frame is an opened Duo's, to be stored turned upright; the run then records the landscape it shows. */
|
||||
const landscapeOnPortrait = (nodes: readonly Record<string, unknown>[], png: Buffer): boolean => {
|
||||
const width = png.readUInt32BE(16), height = png.readUInt32BE(20);
|
||||
const turn = duoScreenshotTurn({ platform, deviceModel: runtime.deviceModel, nodes, width, height });
|
||||
if (turn === "unmatched" && !unmatchedPanel) {
|
||||
unmatchedPanel = true;
|
||||
const root = nodes.find(node => node.type === "Application")?.rect as { width?: unknown; height?: unknown } | undefined;
|
||||
say(`an iPhone Duo frame (${width}×${height} px, root ${root ? `${String(root.width)}×${String(root.height)} pt` : "missing"}) shows neither of its panels; its screenshot is stored as taken, which may be sideways`);
|
||||
}
|
||||
if (turn !== "turn") return false;
|
||||
if (runtime.orientation !== "landscape") { runtime.orientation = "landscape"; if (map.run.environment) map.run.environment.orientation = "landscape"; }
|
||||
return true;
|
||||
};
|
||||
const observe = async (includeImage: boolean): Promise<BlackBoxFrame> => {
|
||||
let nodes: Record<string, unknown>[];
|
||||
let leftApp = false;
|
||||
@@ -411,7 +460,7 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
|
||||
assertDecodedPng(png, "Exploration screenshot");
|
||||
if (png.length > 10 * 1024 * 1024) return { nodes, capturedAt, imageUnavailable: { code: "SCREENSHOT_TOO_LARGE", message: "The screenshot exceeded 10 MB." } };
|
||||
if (solidColour(png)) return { nodes, capturedAt, imageUnavailable: { code: "SCREENSHOT_BLANK", message: BLANK_SCREENSHOT_MESSAGE }, ...(leftApp ? { leftApp } : {}) };
|
||||
return { nodes, capturedAt, frameId: randomUUID(), png, imageSha256: createHash("sha256").update(png).digest("hex"), width: png.readUInt32BE(16), height: png.readUInt32BE(20), ...(leftApp ? { leftApp } : {}) };
|
||||
return { nodes, capturedAt, frameId: randomUUID(), png, imageSha256: createHash("sha256").update(png).digest("hex"), width: png.readUInt32BE(16), height: png.readUInt32BE(20), ...(landscapeOnPortrait(nodes, png) ? { landscape: true } : {}), ...(leftApp ? { leftApp } : {}) };
|
||||
} finally { await rm(path, { force: true }); }
|
||||
};
|
||||
const setupOptions = { command: async (args: string[]) => { const result = controller(guardPositionals([...args, "--json"]), 120_000); return controllerData(result.stdout, args[0] ?? "command"); },
|
||||
@@ -444,7 +493,8 @@ export async function exploreConnectedSession(ctx: ConnectedSessionExploration):
|
||||
}
|
||||
const exploration = blockedAtLaunch ?? await runBlackBoxExploration({ session: model.session, store, map, app: { name: app.name, bundleId, platform, ...(app.version ? { version: app.version } : {}) },
|
||||
inputs, ...(hints ? { hints: hints } : {}), ...(launchUrl ? { launchUrl: launchUrl } : {}), ...(allowDestructive ? { allowDestructive: allowDestructive } : {}), ...(ctx.appIcon ? { appIcon: ctx.appIcon } : {}),
|
||||
deadlineMs, maxScreens: limits.maxScreens, maxActions: limits.maxActions, maxTurns: limits.maxActions * 3 + limits.maxScreens * 2 + 20, now, observe, progress: say,
|
||||
...(model.budgetNote ? { budgetNote: model.budgetNote } : {}),
|
||||
deadlineMs, maxScreens: limits.maxScreens, maxActions: limits.maxActions, maxTurns: turnBudget(limits), now, observe, progress: say,
|
||||
act: async action => {
|
||||
const perform = async () => {
|
||||
if (action.action === "tap" && "frameId" in action) {
|
||||
@@ -508,7 +558,8 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
|
||||
const environment = options.environment ?? process.env;
|
||||
const runner = options.runner ?? defaultRunner;
|
||||
const limits = validatedLimits(options);
|
||||
const inputs = hintInputs(options.hints, environment);
|
||||
const replay = replayPlan(options);
|
||||
const inputs = hintInputs(options.hints, environment, replay);
|
||||
if (options.bundleId !== undefined && !appIdPattern.test(options.bundleId)) throw binaryError("BUNDLE_ID_INVALID", `${options.bundleId} is not a bundle or package id.`, "Pass --bundle-id with the app's bundle identifier (iOS) or package name (Android).");
|
||||
let connection: SimulatorConnection;
|
||||
try { connection = await resolveAttachedSimulatorSession(environment, options.envFile, options.session); }
|
||||
@@ -572,7 +623,7 @@ export async function runSessionExploration(options: SessionExploreOptions): Pro
|
||||
map.run.environment = { deviceModel: device };
|
||||
session = { id: connection.sessionId, platform, device, name: info.name, ...(info.deviceRunSessionUrl ? { url: info.deviceRunSessionUrl } : {}) };
|
||||
|
||||
const model = await preflightModelSession(options, environment, say);
|
||||
const model = replay ? replayModel(replay, platform, limits) : await preflightModelSession(options, environment, say);
|
||||
const outputDirectory = await realpath(await mkdir(options.output, { recursive: true }).then(() => options.output));
|
||||
const store = new ExploreStore(outputDirectory);
|
||||
if (await store.exists()) throw binaryError("OUTPUT_NOT_EMPTY", `${outputDirectory} already holds an Explore map.`, "Pass an empty --output directory.");
|
||||
@@ -665,7 +716,8 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
|
||||
const environment = options.environment ?? process.env;
|
||||
const runner = options.runner ?? defaultRunner;
|
||||
const limits = validatedLimits(options);
|
||||
const inputs = hintInputs(options.hints, environment);
|
||||
const replay = replayPlan(options);
|
||||
const inputs = hintInputs(options.hints, environment, replay);
|
||||
const local = options.simulator === "local";
|
||||
// A local emulator runs this Mac's ABI (arm64-v8a on Apple silicon); EAS's runs x86_64.
|
||||
const emulatorAbi = local ? hostEmulatorAbi() : "x86_64";
|
||||
@@ -682,7 +734,7 @@ export async function runBinaryExploration(options: BinaryExploreOptions): Promi
|
||||
say(`${archive.appName} ${archive.version ?? ""} (${bundleId}, ${platform}, ${(archive.bytes / 1048576).toFixed(0)} MB, sha256 ${archive.sha256.slice(0, 12)})`);
|
||||
if (archive.missingLibs?.length && emulatorAbi === "x86_64") say(`warning: lib/x86_64 lacks ${archive.missingLibs.slice(0, 4).join(", ")}${archive.missingLibs.length > 4 ? ` and ${archive.missingLibs.length - 4} more` : ""} that another ABI ships; the app may stop with UnsatisfiedLinkError on the emulator`);
|
||||
|
||||
const model = await preflightModelSession(options, environment, say);
|
||||
const model = replay ? replayModel(replay, platform, limits) : await preflightModelSession(options, environment, say);
|
||||
|
||||
const outputDirectory = await realpath(await mkdir(options.output, { recursive: true }).then(() => options.output));
|
||||
const store = new ExploreStore(outputDirectory);
|
||||
@@ -871,6 +923,27 @@ export function guardPositionals(args: readonly string[]): string[] {
|
||||
|
||||
const KEYBOARD_RETURN_KEY = /^(return|next|done|go|continue|join)$/i;
|
||||
|
||||
/**
|
||||
* How a frame's screenshot is stored. An opened (or half-open) iPhone Duo lights its inner panel, whose native axes are
|
||||
* portrait, and shows a landscape UI on it: the tree and taps stay on the portrait axes (a 669×951 pt root, the transpose
|
||||
* of `DUO_PANELS.inner`), and a screenshot on those axes is sideways to a person, so it is turned ("turn"). Local
|
||||
* agent-device runs returned exactly that (2007×2853 px). Capture's Argent screenshots on EAS, taken with Argent's
|
||||
* LandscapeRight rotation, came back upright (2853×2007); an image that is already the root's transpose is left as it is.
|
||||
* The root is the full-screen Application node capture takes its coordinate space from, matched to within a pixel and a
|
||||
* point. A Duo frame on neither panel is "unmatched": it is stored as taken, and the caller says so. An empty tree, read
|
||||
* mid-draw, has nothing to judge.
|
||||
*/
|
||||
export function duoScreenshotTurn(frame: { platform: Platform; deviceModel: string; nodes: readonly Record<string, unknown>[]; width: number; height: number }): "turn" | "keep" | "unmatched" {
|
||||
if (frame.platform !== "ios" || !/\bduo\b/i.test(frame.deviceModel) || !frame.nodes.length) return "keep";
|
||||
const space = screenSpace(frame.nodes, frame, 1);
|
||||
if (!space) return "unmatched";
|
||||
const near = (value: number, points: number) => Math.abs(value - points) < 1;
|
||||
const portraitAxes = near(space.width, DUO_PANELS.inner.height) && near(space.height, DUO_PANELS.inner.width);
|
||||
if (portraitAxes && !space.transposed && frame.width < frame.height) return "turn";
|
||||
const panel = Object.values(DUO_PANELS).some(({ width, height }) => (near(space.width, width) && near(space.height, height)) || (near(space.width, height) && near(space.height, width)));
|
||||
return panel ? "keep" : "unmatched";
|
||||
}
|
||||
|
||||
/** A node that is not the app's UI: another package drew it (Android's system UI over the app), or it is Explore's own route marker, which an app
|
||||
* instrumented for PR runs renders as a 2x2 hidden text. */
|
||||
export function foreignNode(node: Record<string, unknown>, bundleId: string): boolean {
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
* injected, so the translation is tested against recorded Argent output.
|
||||
*/
|
||||
import { CONTROLLER_SESSION } from "./common.js";
|
||||
import { INPUT_METHOD_PACKAGES, KEYBOARD_WINDOW_IDENTIFIERS } from "./keyboard.js";
|
||||
import type { DecodedFrame } from "../shared/png-evidence.js";
|
||||
|
||||
export type ArgentPlatform = "ios" | "android";
|
||||
@@ -88,7 +89,7 @@ export function parseCommand(argv: readonly string[]): { positional: string[]; f
|
||||
}
|
||||
|
||||
type ArgentNode = { role?: string; type?: string; label?: string; identifier?: string; value?: string; contentDescription?: string; disabled?: boolean;
|
||||
frame?: Rect; traits?: string[]; hidden?: boolean; selected?: boolean; password?: boolean; bundleId?: string; children?: ArgentNode[] };
|
||||
frame?: Rect; traits?: string[]; hidden?: boolean; selected?: boolean; focused?: boolean; password?: boolean; bundleId?: string; children?: ArgentNode[] };
|
||||
export type ArgentTree = { schemaVersion?: number; source?: string; screen?: { width: number; height: number }; roots?: ArgentNode[];
|
||||
truncated?: boolean; alertVisible?: boolean; keyboardVisible?: boolean; foregroundApp?: string };
|
||||
|
||||
@@ -137,7 +138,11 @@ export function snapshotFromTree(tree: ArgentTree, platform: ArgentPlatform, gen
|
||||
if (root || !node.frame || ![node.frame.x, node.frame.y, node.frame.width, node.frame.height].every(finite)) return clip;
|
||||
return clip && intersect(node.frame, clip);
|
||||
};
|
||||
const visit = (node: ArgentNode, depth: number, parentIndex: number | undefined, appRoot: boolean, clip: Rect | undefined, root: boolean) => {
|
||||
const keyboard = platform === "ios" ? keyboardWindows(roots) : new Set<ArgentNode>();
|
||||
/** `owner` marks every node of a window that is the system keyboard's, as agent-device marks an input method's nodes. */
|
||||
type Owner = { bundleId?: string; systemKeyboard?: true };
|
||||
const visit = (node: ArgentNode, depth: number, parentIndex: number | undefined, appRoot: boolean, clip: Rect | undefined, root: boolean, owner: Owner) => {
|
||||
if (keyboard.has(node)) owner = { ...owner, systemKeyboard: true };
|
||||
const frame = node.frame;
|
||||
const inner = clipOf(node, clip, root);
|
||||
const rect = frame && [frame.x, frame.y, frame.width, frame.height].every(finite) ? toPoints(frame) : undefined;
|
||||
@@ -159,19 +164,42 @@ export function snapshotFromTree(tree: ArgentTree, platform: ArgentPlatform, gen
|
||||
rect: appRoot ? { x: 0, y: 0, width: screen.width, height: screen.height } : rect,
|
||||
...(label ? { label } : {}), ...(node.identifier ? { identifier: node.identifier } : {}),
|
||||
...(node.value !== undefined ? { value: node.value } : {}), enabled, hittable: enabled && visible && !appRoot,
|
||||
...(visible ? {} : { visible: false }), ...(node.selected || traits.includes("selected") ? { selected: true } : {}) });
|
||||
...(visible ? {} : { visible: false }), ...(node.selected || traits.includes("selected") ? { selected: true } : {}),
|
||||
// The field with the keyboard focus, as agent-device marks it, so capture can wait for the keyboard it raises.
|
||||
...(node.focused === true || traits.includes("isEditing") ? { focused: true } : {}),
|
||||
...owner });
|
||||
refs[ref] = { rect: appRoot ? { x: 0, y: 0, width: screen.width, height: screen.height } : rect, type,
|
||||
...(node.value !== undefined ? { value: node.value } : {}), ...(type === "SecureTextField" ? { secure: true } : {}) };
|
||||
}
|
||||
for (const child of node.children ?? []) visit(child, depth + 1, index, false, inner, false);
|
||||
for (const child of node.children ?? []) visit(child, depth + 1, index, false, inner, false, owner);
|
||||
};
|
||||
for (const root of roots) {
|
||||
const isAppRoot = platform === "ios" ? root.type === "Application" || nodes.length === 0 : root.bundleId === app;
|
||||
visit(root, 0, undefined, isAppRoot, { x: 0, y: 0, width: 1, height: 1 }, true);
|
||||
// An Android input method's window names its package on each node, as agent-device's do, which marks the keyboard.
|
||||
const inputMethod = platform === "android" && root.bundleId && INPUT_METHOD_PACKAGES.has(root.bundleId) ? root.bundleId : undefined;
|
||||
visit(root, 0, undefined, isAppRoot, { x: 0, y: 0, width: 1, height: 1 }, true, inputMethod ? { bundleId: inputMethod } : {});
|
||||
}
|
||||
return { nodes, refs, screen: { width: screen.width, height: screen.height } };
|
||||
}
|
||||
|
||||
/**
|
||||
* The windows iOS draws the software keyboard in. Argent lists each window as a child of its root, the app's first. A
|
||||
* later window is the keyboard's when it holds XCTest's `Keyboard` element, which no app can give itself; an identifier
|
||||
* never decides it alone, since an app can set any identifier on its own views (a second window of the app's with a
|
||||
* `testID="inputView"` stays the app's). While such a window is up, UIKit also hosts the keyboard's input view
|
||||
* placeholder in a window of its own: a later window whose only named node is an `inputView` belongs to it too.
|
||||
*/
|
||||
function keyboardWindows(roots: readonly ArgentNode[]): Set<ArgentNode> {
|
||||
const holds = (node: ArgentNode, test: (node: ArgentNode) => boolean): boolean => test(node) || (node.children ?? []).some(child => holds(child, test));
|
||||
const later = roots.flatMap(root => (root.children ?? []).slice(1));
|
||||
const keyboard = later.filter(window => holds(window, node => node.type === "Keyboard"));
|
||||
if (!keyboard.length) return new Set();
|
||||
const named = (node: ArgentNode) => Boolean(node.label?.trim() || node.contentDescription?.trim() || node.value !== undefined
|
||||
|| (node.identifier && !KEYBOARD_WINDOW_IDENTIFIERS.has(node.identifier)));
|
||||
const placeholders = later.filter(window => !keyboard.includes(window) && holds(window, node => node.identifier === "inputView") && !holds(window, named));
|
||||
return new Set([...keyboard, ...placeholders]);
|
||||
}
|
||||
|
||||
function intersect(a: Rect, b: Rect): Rect | undefined {
|
||||
const x = Math.max(a.x, b.x), y = Math.max(a.y, b.y);
|
||||
const right = Math.min(a.x + a.width, b.x + b.width), bottom = Math.min(a.y + a.height, b.y + b.height);
|
||||
@@ -438,7 +466,9 @@ export async function runBridgeCommand(argv: readonly string[], context: ArgentB
|
||||
converted.nodes.push({ depth: 1, index: converted.nodes.length, parentIndex: 0, type: "Alert", ref: `e${converted.nodes.length + 1}`,
|
||||
rect: { x: 0, y: 0, width: converted.screen.width, height: converted.screen.height }, label: "System alert", enabled: true, hittable: false });
|
||||
}
|
||||
// Whether a keyboard shows, which Argent's Android tree does not hold: the input method's window is not in it.
|
||||
return success({ nodes: converted.nodes, truncated: current.truncated === true, refsGeneration: state.generation,
|
||||
...(typeof current.keyboardVisible === "boolean" ? { keyboardVisible: current.keyboardVisible } : {}),
|
||||
...(current.foregroundApp ? { appBundleId: current.foregroundApp, appName: current.foregroundApp } : {}),
|
||||
snapshotQuality: { state: "healthy", backend: "argent" } });
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import type {
|
||||
ElementRole,
|
||||
InputElement,
|
||||
} from "../types.js";
|
||||
import { INPUT_METHOD_PACKAGES, keyboardNodes } from "./keyboard.js";
|
||||
|
||||
export type ControllerParseResult = {
|
||||
controller: Controller;
|
||||
@@ -221,10 +222,7 @@ function nodeToElement(node: Record<string, unknown>): InputElement | undefined
|
||||
const SYSTEM_UI_PACKAGES = new Set([
|
||||
"android",
|
||||
"com.android.systemui",
|
||||
"com.google.android.inputmethod.latin",
|
||||
"com.samsung.android.honeyboard",
|
||||
"com.touchtype.swiftkey",
|
||||
"com.microsoft.swiftkey",
|
||||
...INPUT_METHOD_PACKAGES,
|
||||
]);
|
||||
|
||||
export function isSystemUiNode(node: Record<string, unknown>): boolean {
|
||||
@@ -240,6 +238,12 @@ function collectJsonElements(value: unknown, output: InputElement[], seen: Set<u
|
||||
return;
|
||||
}
|
||||
const record = value as Record<string, unknown>;
|
||||
// iOS's software keyboard is system UI too, like Android's: its keys and its Emoji and Dictate buttons are not the
|
||||
// app's elements. A keyboard that rose on one side only made "Added button “Dictate”" lead a review whose real
|
||||
// change was a field's label (brentvatne/euxy#172, review run 39362642343).
|
||||
if (Array.isArray(record.nodes)) {
|
||||
for (const node of keyboardNodes(record.nodes.filter((node): node is Record<string, unknown> => !!node && typeof node === "object" && !Array.isArray(node)))) seen.add(node);
|
||||
}
|
||||
const element = isSystemUiNode(record) ? undefined : nodeToElement(record);
|
||||
if (element) output.push(element);
|
||||
for (const [key, child] of Object.entries(record)) {
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
/**
|
||||
* The system software keyboard in a controller snapshot (agent-device's flat `nodes`, which the Argent bridge also
|
||||
* returns), and how long capture waits for it. Recognition is deliberately narrow: dropping one of the app's own elements
|
||||
* hides a real change, which is worse than the keyboard noise this removes. A node is the keyboard's only when it, or a
|
||||
* node above it, provably belongs to the system keyboard; nothing is inferred from where an element sits on the screen,
|
||||
* and a `Key` alone proves nothing (React Native's `accessibilityRole="keyboardkey"` gives an app's PIN pad that type).
|
||||
*/
|
||||
|
||||
type Rect = { x: number; y: number; width: number; height: number };
|
||||
type SnapshotNode = Record<string, unknown>;
|
||||
|
||||
/** Android input methods: their windows name the IME's package, not the app's. */
|
||||
export const INPUT_METHOD_PACKAGES: ReadonlySet<string> = new Set([
|
||||
"com.google.android.inputmethod.latin",
|
||||
"com.android.inputmethod.latin",
|
||||
"com.samsung.android.honeyboard",
|
||||
"com.touchtype.swiftkey",
|
||||
"com.microsoft.swiftkey",
|
||||
]);
|
||||
|
||||
/**
|
||||
* UIKit's identifiers for the input view placeholder and the prediction bar. They mark a window as the keyboard's (see
|
||||
* the Argent bridge's `keyboardWindows`); inside the app's own window an element with one of them is the app's.
|
||||
*/
|
||||
export const KEYBOARD_WINDOW_IDENTIFIERS: ReadonlySet<string> = new Set(["inputView", "SystemInputAssistantView"]);
|
||||
|
||||
/**
|
||||
* The keyboard's native view classes, which agent-device's host accessibility bridge reports as `role`. The dock's Emoji
|
||||
* and Dictate buttons (UIKeyboardDockItemButton) can sit outside the key plane's bounds, so they are named here.
|
||||
*/
|
||||
const KEYBOARD_VIEW_CLASSES = new Set(["UIKeyboardDockItemButton", "UIAccessibilityElementKBKey", "UIKeyboardLayoutStar",
|
||||
"TUIPredictionView", "TUIPredictionViewCell", "_UIRemoteKeyboardPlaceholderView", "TUISystemInputAssistantView",
|
||||
"TUISystemInputAssistantPageView"]);
|
||||
|
||||
const EDITABLE_TYPES = new Set(["textfield", "securetextfield", "searchfield", "textview"]);
|
||||
|
||||
const text = (value: unknown) => typeof value === "string" ? value : "";
|
||||
|
||||
function rectOf(node: SnapshotNode): Rect | undefined {
|
||||
const rect = node.rect as Partial<Rect> | undefined;
|
||||
return rect && [rect.x, rect.y, rect.width, rect.height].every(Number.isFinite) && rect.width! > 0 && rect.height! > 0 ? rect as Rect : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* A node that belongs to the system keyboard by what it is, not where it is: XCTest's `Keyboard` element (both
|
||||
* controllers), a node of a window the Argent bridge found to be the keyboard's (`systemKeyboard`), one of the keyboard's
|
||||
* native view classes, or a node of an Android input method's window.
|
||||
*/
|
||||
function keyboardRoot(node: SnapshotNode): boolean {
|
||||
return node.systemKeyboard === true || text(node.type).toLowerCase() === "keyboard" || KEYBOARD_VIEW_CLASSES.has(text(node.role))
|
||||
|| INPUT_METHOD_PACKAGES.has(text(node.bundleId ?? node.packageName ?? node.package));
|
||||
}
|
||||
|
||||
/** Every node that belongs to the system keyboard: those `keyboardRoot` names and the nodes beneath them. */
|
||||
export function keyboardNodes(nodes: readonly SnapshotNode[]): Set<SnapshotNode> {
|
||||
const owned = new Set<SnapshotNode>();
|
||||
if (!nodes.some(keyboardRoot)) return owned;
|
||||
// Only an explicit parent links a node to a keyboard above it: a node without one is no node's descendant.
|
||||
const byIndex = new Map(nodes.filter(node => typeof node.index === "number").map(node => [node.index as number, node]));
|
||||
for (const node of nodes) {
|
||||
const seen = new Set<SnapshotNode>();
|
||||
for (let current: SnapshotNode | undefined = node; current && !seen.has(current);
|
||||
current = typeof current.parentIndex === "number" ? byIndex.get(current.parentIndex) : undefined) {
|
||||
seen.add(current);
|
||||
if (keyboardRoot(current)) { owned.add(node); break; }
|
||||
}
|
||||
}
|
||||
return owned;
|
||||
}
|
||||
|
||||
function union(rects: readonly Rect[]): Rect | undefined {
|
||||
if (!rects.length) return undefined;
|
||||
const x = Math.min(...rects.map(rect => rect.x)), y = Math.min(...rects.map(rect => rect.y));
|
||||
return { x, y, width: Math.max(...rects.map(rect => rect.x + rect.width)) - x, height: Math.max(...rects.map(rect => rect.y + rect.height)) - y };
|
||||
}
|
||||
|
||||
/**
|
||||
* Where the visible keyboard is, from the nodes that span it (the `Keyboard` element, the keyboard window's input view and
|
||||
* prediction bar, the native views, an input method's window), or undefined when none shows. Containers inside the
|
||||
* keyboard can report a scroll extent past the screen, so not every node it owns counts.
|
||||
*/
|
||||
export function keyboardBounds(nodes: readonly SnapshotNode[]): Rect | undefined {
|
||||
return union(nodes.filter(node => node.visible !== false && keyboardRoot(node) && (node.systemKeyboard !== true
|
||||
|| text(node.type).toLowerCase() === "keyboard" || KEYBOARD_WINDOW_IDENTIFIERS.has(text(node.identifier)))).flatMap(node => rectOf(node) ?? []));
|
||||
}
|
||||
|
||||
/**
|
||||
* A text input that has the keyboard focus: agent-device reports `focused` from XCTest's keyboard focus and Android's
|
||||
* focused flag; the Argent bridge passes on Argent's own.
|
||||
*/
|
||||
export function focusedTextInput(nodes: readonly SnapshotNode[]): boolean {
|
||||
return nodes.some(node => node.focused === true && node.visible !== false
|
||||
&& (EDITABLE_TYPES.has(text(node.type).toLowerCase()) || /EditText$/.test(text(node.type)) || node.editable === true));
|
||||
}
|
||||
|
||||
/** A snapshot as the wait reads it: Argent also says whether a keyboard shows, which its Android tree cannot. */
|
||||
export type KeyboardRead = { nodes: readonly SnapshotNode[]; keyboardVisible?: boolean };
|
||||
|
||||
/** Where the keyboard is in one read: its bounds, or only that one shows when the controller cannot place it. */
|
||||
function keyboardPosition(read: KeyboardRead): string | undefined {
|
||||
const bounds = keyboardBounds(read.nodes);
|
||||
return bounds ? JSON.stringify(bounds) : read.keyboardVisible === true ? "visible" : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* How capture waits for a focused field's keyboard. Reads are 250 ms apart at most, and at least two follow the first,
|
||||
* so a controller whose snapshot takes a second or more still sees the keyboard rise. Waiting for a keyboard that has not
|
||||
* appeared ends after a second; one that has appeared but has not yet held still is read until it does, up to six reads
|
||||
* or four seconds, so a slow controller cannot keep the capture waiting.
|
||||
*/
|
||||
export const KEYBOARD_POLL_MS = 250;
|
||||
const KEYBOARD_WAIT_MS = 1_000;
|
||||
const KEYBOARD_CAP_MS = 4_000;
|
||||
const KEYBOARD_MIN_READS = 2;
|
||||
const KEYBOARD_MAX_READS = 6;
|
||||
|
||||
export type KeyboardSettle = {
|
||||
/** `shown`: in the same place in two reads running. `unfocused`: the field let go and no keyboard shows. */
|
||||
readonly outcome: "shown" | "moving" | "absent" | "unfocused";
|
||||
readonly reads: number;
|
||||
/** Whether to read the screen again, `elapsedMs` after the first read. */
|
||||
more(elapsedMs: number): boolean;
|
||||
read(next: KeyboardRead): void;
|
||||
};
|
||||
|
||||
/**
|
||||
* The wait for a screen whose text field has the focus, or undefined when none has it. A keyboard already up counts only
|
||||
* once a later read finds it in the same place: one caught rising or going down is still moving.
|
||||
*/
|
||||
export function keyboardSettle(first: KeyboardRead): KeyboardSettle | undefined {
|
||||
if (!focusedTextInput(first.nodes)) return undefined;
|
||||
let previous = keyboardPosition(first), reads = 0;
|
||||
let outcome: KeyboardSettle["outcome"] = previous ? "moving" : "absent";
|
||||
return {
|
||||
get outcome() { return outcome; },
|
||||
get reads() { return reads; },
|
||||
more: elapsedMs => (outcome === "moving" || outcome === "absent") && reads < KEYBOARD_MAX_READS
|
||||
&& (reads < KEYBOARD_MIN_READS || (elapsedMs < KEYBOARD_CAP_MS && (outcome === "moving" || elapsedMs < KEYBOARD_WAIT_MS))),
|
||||
read: next => {
|
||||
reads += 1;
|
||||
const position = keyboardPosition(next);
|
||||
outcome = position && position === previous ? "shown" : position ? "moving" : focusedTextInput(next.nodes) ? "absent" : "unfocused";
|
||||
previous = position;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -17,6 +17,7 @@ import { type CaptureSetupSnapshot, credentialSavePromptDecline, describeSetupAc
|
||||
import { silentProgress } from "../shared/progress.js";
|
||||
import type { CaptureSetupReady } from "../config/explore-config.js";
|
||||
import { READINESS_POLL_MS, defaultReadinessSeconds } from "./readiness.js";
|
||||
import { KEYBOARD_POLL_MS, keyboardSettle, type KeyboardRead, type KeyboardSettle } from "./keyboard.js";
|
||||
import { CAPTURE_OBSERVER_IDENTIFIER, captureObserverMarkers, parseCaptureDestination, captureDestinationMatches } from "./observer.js";
|
||||
import { type CiAgentRuntimeOptions, runCiAgentNavigation, assertAgentCaptureContext } from "./agent/runtime.js";
|
||||
import { createExplorationInputs } from "../config/exploration-inputs.js";
|
||||
@@ -92,6 +93,11 @@ function snapshotNodes(raw: string): Record<string, unknown>[] {
|
||||
return data.nodes;
|
||||
}
|
||||
|
||||
/** Whether the controller says a keyboard shows: Argent does, and its Android tree has no input method window to show it. */
|
||||
function keyboardVisibility(data: Record<string, unknown>): { keyboardVisible?: boolean } {
|
||||
return typeof data.keyboardVisible === "boolean" ? { keyboardVisible: data.keyboardVisible } : {};
|
||||
}
|
||||
|
||||
/** The visible element each required label matched, so an image assessor can find icon-only controls. */
|
||||
export function verifiedCaptureLabels(nodes: readonly Record<string, unknown>[], requiredLabels: readonly string[],
|
||||
/** The screen's portrait height, when the screenshot is that screen turned a quarter clockwise (an opened iPhone Duo). */
|
||||
@@ -106,6 +112,24 @@ export function verifiedCaptureLabels(nodes: readonly Record<string, unknown>[],
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* The full-screen logical space a screenshot shows: the one Application node at the origin whose aspect is the image's,
|
||||
* at any pixel density. Failing that, the one whose aspect is the image's transpose (`transposed`): the image is then the
|
||||
* screen turned a quarter. A square screen matches both ways and is read upright. `slack` is how many image pixels the
|
||||
* root, scaled to the image, may miss it by; capture asks for an exact match.
|
||||
*/
|
||||
export function screenSpace(nodes: readonly Record<string, unknown>[], image: { width: number; height: number }, slack = 0): { width: number; height: number; transposed: boolean } | undefined {
|
||||
const roots = (transposed: boolean) => nodes.filter(node => node.type === "Application" && isRecord(node.rect)
|
||||
&& node.rect.x === 0 && node.rect.y === 0 && typeof node.rect.width === "number" && typeof node.rect.height === "number"
|
||||
&& Number.isFinite(node.rect.width) && Number.isFinite(node.rect.height) && node.rect.width > 0 && node.rect.height > 0
|
||||
&& Math.abs(transposed ? node.rect.width * image.width - node.rect.height * image.height : node.rect.width * image.height - node.rect.height * image.width) <= slack * node.rect.height);
|
||||
const upright = roots(false), turned = upright.length ? [] : roots(true);
|
||||
const matched = upright.length ? upright : turned;
|
||||
if (matched.length !== 1) return undefined;
|
||||
const rect = matched[0]!.rect as { width: number; height: number };
|
||||
return { width: rect.width, height: rect.height, transposed: turned.length === 1 };
|
||||
}
|
||||
|
||||
/**
|
||||
* A swipe that closes a presented sheet, or undefined when the snapshot does not prove a sheet.
|
||||
* The route observer is drawn at the focused scene's top-left corner. A sheet's scene starts below
|
||||
@@ -377,7 +401,7 @@ export async function capturePlannedRoutes(options: {
|
||||
lastSnapshot = raw;
|
||||
const generation = typeof data.refsGeneration === "number" ? data.refsGeneration : undefined;
|
||||
const decline = declined < 2 && generation ? credentialSavePromptDecline(nodes) : undefined;
|
||||
if (!decline) return { nodes, truncated: false, ...(generation ? { refsGeneration: generation } : {}) };
|
||||
if (!decline) return { nodes, truncated: false, ...(generation ? { refsGeneration: generation } : {}), ...keyboardVisibility(data) };
|
||||
controller(["press", `@${String(decline.ref).replace(/^@/, "")}~s${generation}`, "--level", "full", "--json"]);
|
||||
recordDeclinedPrompt("deterministic");
|
||||
controller(["wait", "500", "--level", "full", "--json"]);
|
||||
@@ -507,6 +531,35 @@ export async function capturePlannedRoutes(options: {
|
||||
.sort());
|
||||
/** The controller's own quiescence gate; a timeout is reported by the snapshot comparison below. */
|
||||
const settle = () => { controller(["wait", "stable", "500", "8000", "--level", "full", "--json"], true); };
|
||||
/**
|
||||
* A screen that focused a text field shows the keyboard that field raises, as its users see it. The keyboard rises
|
||||
* after the screen has settled, so the controller's quiescence gate can pass in between: in brentvatne/euxy#172
|
||||
* (review run 39362642343) the rename sheet's base was captured with its cursor showing and no keyboard, its head with
|
||||
* the keyboard up, and the 48% mismatch was the keyboard. The screen is read again until the keyboard is in the same
|
||||
* place twice running (see keyboardSettle); a keyboard that never comes is not waited for long, and none is dismissed.
|
||||
*/
|
||||
const awaitKeyboard = (route: SourceRoute, first: KeyboardRead): readonly Record<string, unknown>[] => {
|
||||
const settle = keyboardSettle(first);
|
||||
if (!settle) return first.nodes;
|
||||
const startedAt = clock();
|
||||
let current = first, pollStartedAt = startedAt;
|
||||
// A read counts for at least one poll interval, so a clock that stands still (a fake controller) still expires.
|
||||
const elapsed = () => Math.max(clock() - startedAt, settle.reads * KEYBOARD_POLL_MS);
|
||||
while (settle.more(elapsed())) {
|
||||
(capture.sleep ?? pause)(KEYBOARD_POLL_MS - (clock() - pollStartedAt));
|
||||
pollStartedAt = clock();
|
||||
current = snapshot();
|
||||
settle.read(current);
|
||||
}
|
||||
recordKeyboardWait(route.id, settle, elapsed());
|
||||
return current.nodes;
|
||||
};
|
||||
const recordKeyboardWait = (route: string | undefined, settle: KeyboardSettle, elapsedMs: number) => {
|
||||
const diagnostics = join(store.root, "capture-diagnostics");
|
||||
mkdirSync(diagnostics, { recursive: true, mode: 0o700 });
|
||||
appendFileSync(join(diagnostics, "keyboard.jsonl"), `${JSON.stringify({ ...(route ? { route } : { phase: "agent-observe" }), outcome: settle.outcome,
|
||||
reads: settle.reads, elapsedMs, at: new Date(clock()).toISOString() })}\n`, { mode: 0o600 });
|
||||
};
|
||||
const attempted = new Set<string>();
|
||||
const captureCurrent = async (route: SourceRoute, attempt: number, before?: readonly Record<string, unknown>[],
|
||||
unattributed?: NonNullable<CaptureBundle["outcomes"][string]["observed"]> & { message: string }) => {
|
||||
@@ -514,9 +567,11 @@ export async function capturePlannedRoutes(options: {
|
||||
attempted.add(route.id);
|
||||
const temporary = await mkdtemp(join(store.root, ".capture-"));
|
||||
try {
|
||||
const accepted = before ?? waitForDestination(route, () => resolveDeepLinkConfirmation());
|
||||
if (unattributed) assertNoAlert({ route, nodes: accepted });
|
||||
else landing(accepted, route, inventory);
|
||||
const arrived = before ?? waitForDestination(route, () => resolveDeepLinkConfirmation());
|
||||
if (unattributed) assertNoAlert({ route, nodes: arrived });
|
||||
else landing(arrived, route, inventory);
|
||||
const accepted = unattributed ? arrived : awaitKeyboard(route, { nodes: arrived });
|
||||
if (accepted !== arrived) landing(accepted, route, inventory);
|
||||
assertSafeScreenshot(accepted);
|
||||
const screenshot = join(temporary, "screen.png");
|
||||
await writeFile(screenshot, "", { flag: "wx", mode: 0o600 });
|
||||
@@ -663,7 +718,7 @@ export async function capturePlannedRoutes(options: {
|
||||
const generation = typeof data.refsGeneration === "number" ? data.refsGeneration : undefined;
|
||||
const decline = declined < 2 && generation ? credentialSavePromptDecline(nodes) : undefined;
|
||||
if (!decline) {
|
||||
const snapshot: CaptureSetupSnapshot = { nodes, truncated: false, ...(generation ? { refsGeneration: generation } : {}) };
|
||||
const snapshot: CaptureSetupSnapshot = { nodes, truncated: false, ...(generation ? { refsGeneration: generation } : {}), ...keyboardVisibility(data) };
|
||||
return { snapshot, raw };
|
||||
}
|
||||
await command(["press", `@${String(decline.ref).replace(/^@/, "")}~s${generation}`, "--level", "full", "--json"], signal);
|
||||
@@ -693,18 +748,11 @@ export async function capturePlannedRoutes(options: {
|
||||
* then the transpose of the frame, and a tap is turned back (see `toLogical`).
|
||||
*/
|
||||
const bounds = (frame: Candidate) => {
|
||||
const roots = (transposed: boolean) => frame.snapshot.nodes.filter(node => node.type === "Application" && isRecord(node.rect)
|
||||
&& node.rect.x === 0 && node.rect.y === 0 && typeof node.rect.width === "number" && typeof node.rect.height === "number"
|
||||
&& Number.isFinite(node.rect.width) && Number.isFinite(node.rect.height) && node.rect.width > 0 && node.rect.height > 0
|
||||
&& (transposed ? node.rect.width * frame.width === node.rect.height * frame.height : node.rect.width * frame.height === node.rect.height * frame.width));
|
||||
// A square screen matches both ways; it is read upright, as before rotated frames existed.
|
||||
const upright = roots(false), turned = upright.length ? [] : roots(true);
|
||||
const matched = upright.length ? upright : turned;
|
||||
if (matched.length !== 1) throw captureError("AGENT_COORDINATE_BOUNDS_UNVERIFIED", "The screenshot does not identify one matching full-screen logical coordinate space.");
|
||||
// The official iOS screenshot is full-screen. Its default is 1x, but a proven
|
||||
// uniform pixel density must still map to XCTest points rather than raw pixels.
|
||||
const rect = matched[0]!.rect as { width: number; height: number };
|
||||
return { width: rect.width, height: rect.height, transposed: turned.length === 1 };
|
||||
const space = screenSpace(frame.snapshot.nodes, frame);
|
||||
if (!space) throw captureError("AGENT_COORDINATE_BOUNDS_UNVERIFIED", "The screenshot does not identify one matching full-screen logical coordinate space.");
|
||||
return space;
|
||||
};
|
||||
/**
|
||||
* A normalized point (x, y) in the screenshot, in the tree's logical points. Upright, it scales by the root. Turned:
|
||||
@@ -769,16 +817,32 @@ export async function capturePlannedRoutes(options: {
|
||||
}
|
||||
}
|
||||
};
|
||||
/** The same wait for a focused field's keyboard as a route link's capture, before an observation's screenshot. */
|
||||
const awaitKeyboardAsync = async (first: CaptureSetupSnapshot, signal: AbortSignal): Promise<CaptureSetupSnapshot> => {
|
||||
const settle = keyboardSettle(first);
|
||||
if (!settle) return first;
|
||||
const startedAt = clock();
|
||||
let current = first;
|
||||
const elapsed = () => Math.max(clock() - startedAt, settle.reads * KEYBOARD_POLL_MS);
|
||||
while (settle.more(elapsed())) {
|
||||
await sleep(KEYBOARD_POLL_MS, undefined, { signal: liveSignal(signal) });
|
||||
current = (await readSnapshot(signal)).snapshot;
|
||||
settle.read(current);
|
||||
}
|
||||
recordKeyboardWait(undefined, settle, elapsed());
|
||||
return current;
|
||||
};
|
||||
const observe = async (signal: AbortSignal, options?: { includeImage: boolean }) => {
|
||||
await discard();
|
||||
await appIdentity(signal);
|
||||
const first = await readSnapshotThroughDeferral(signal);
|
||||
const before = first.snapshot;
|
||||
let before = first.snapshot;
|
||||
if (first.retries > 0) {
|
||||
await appendFile(join(dirname(tracePath), "accessibility-deferral.jsonl"), `${JSON.stringify({ retries: first.retries, waitedMs: first.deferredMs,
|
||||
at: new Date(clock()).toISOString() })}\n`, { mode: 0o600 });
|
||||
}
|
||||
if (options?.includeImage === false) return { snapshot: before };
|
||||
before = await awaitKeyboardAsync(before, signal);
|
||||
try { assertSafeScreenshot(before.nodes); }
|
||||
catch (error) {
|
||||
if (error instanceof CiError && error.code === "CAPTURE_SENSITIVE_INPUT") return { snapshot: before, imageUnavailable: { code: error.code, message: error.message } };
|
||||
|
||||
+27
-3
@@ -360,9 +360,10 @@ async function executeCiMap(options: CiMapOptions, setStage: (stage: string) =>
|
||||
let captureFailure: unknown;
|
||||
const finish = async (): Promise<CiMapResult> => {
|
||||
if (!(await store.exists())) await store.initialize(map, false);
|
||||
// A session making only secondary passes owns the `<id>@<scheme>` frames of those passes, not the primary ones.
|
||||
// A session making only secondary passes owns the `<id>@<scheme>` frames of those passes, not the primary ones, even
|
||||
// when it could not switch to them and so captured none.
|
||||
const ownFrameIds = options.appearancePass && !options.appearancePass.capture.includes(colorSchemes[0]!)
|
||||
? selectedIds.flatMap(id => colorSchemes.filter(scheme => options.appearancePass!.capture.includes(scheme)).map(scheme => schemeScreenId(id, scheme)))
|
||||
? selectedIds.flatMap(id => options.appearancePass!.capture.filter(scheme => scheme !== colorSchemes[0]).map(scheme => schemeScreenId(id, scheme)))
|
||||
: selectedIds;
|
||||
const incomplete = captureFailure !== undefined || ownFrameIds.some(id => outcomes[id]?.status !== "captured" || outcomes[id]?.landing !== "verified")
|
||||
|| map.gaps.some(gap => gap.status === "open");
|
||||
@@ -751,8 +752,28 @@ async function executeCiMap(options: CiMapOptions, setStage: (stage: string) =>
|
||||
if (!handoff.connection) throw new Error("Simulator handoff is missing its controller connection");
|
||||
connection = handoff.connection;
|
||||
await rm(dotenvPath, { force: true });
|
||||
if (options.platform === "ios") controlActions = simulatorControlActions(sessionEas, sessionId);
|
||||
if (options.platform === "ios") controlActions = simulatorControlActions(sessionEas, sessionId, controlOptions);
|
||||
if (!controlActions.has("appearance.set")) narrowToArgentScheme();
|
||||
// A side split over sessions gives this one only some of its appearances (captureSideInSessions). One that cannot
|
||||
// switch to any of them has nothing to capture, so it stops here, before the app is installed or the device folded:
|
||||
// in brentvatne/euxy's main baseline run 39487618529, the dark session folded the device for frames it would never
|
||||
// take, and its failed fold failed the target that the light session had captured.
|
||||
const pass = options.appearancePass;
|
||||
if (pass && !colorSchemes.some(scheme => pass.capture.includes(scheme))) {
|
||||
const skipped = pass.capture.join(", ");
|
||||
// The export merge needs the primary appearance's frames; a session that cannot take them fails the side.
|
||||
if (pass.capture.includes(pass.all[0]!)) throw new CiError({ code: "SESSION_APPEARANCE_UNAVAILABLE", stage: "session", retryable: true,
|
||||
message: `Simulator session ${sessionId} cannot switch to ${skipped}, the primary appearance it was given to capture.`,
|
||||
nextAction: "Rerun the capture on a new session, or set capture.appearance to the appearance this controller renders." });
|
||||
say(`session ${sessionId} cannot switch to ${skipped}, the appearance it was given; it stops without capturing, and the report has no ${skipped} frames`);
|
||||
const error = { code: "SESSION_APPEARANCE_UNAVAILABLE", stage: "session", retryable: true,
|
||||
message: `Simulator session ${sessionId} could not switch to ${skipped}.`, nextAction: "Rerun the capture on a new session." };
|
||||
for (const id of selectedIds) for (const scheme of pass.capture) {
|
||||
outcomes[schemeScreenId(id, scheme)] = { status: "failed", attempts: 0, landing: "unverified", reason: error.message, error };
|
||||
}
|
||||
// The finally block below stops the session after the export.
|
||||
return await finish();
|
||||
}
|
||||
}
|
||||
const controllerConfig = join(simulatorProject, "agent-device.json");
|
||||
await writeFile(controllerConfig, "{}\n", { flag: "wx", mode: 0o400 });
|
||||
@@ -825,6 +846,9 @@ async function executeCiMap(options: CiMapOptions, setStage: (stage: string) =>
|
||||
// can leave the panel it lights black, as it does for Argent: the controller then reads the old panel and a
|
||||
// relaunch draws nothing. The display is the check, through the controller's own screenshot: in the pose's shape
|
||||
// and drawn. If it is not, SpringBoard is restarted, the app reopened and the device folded again, once.
|
||||
// The open pose's shape is taken to be landscape. That is unverified for agent-device on EAS: local agent-device
|
||||
// runs returned the opened inner panel on its portrait axes (2007×2853), as explore:binary turns it from; the
|
||||
// landscape shots seen on EAS (2853×2007) were Argent's, taken with its LandscapeRight rotation.
|
||||
const open = (relaunch: boolean) => runner({ command: controllerCommand, args: ["open", options.bundleId, ...(relaunch ? ["--relaunch"] : []), "--session", CONTROLLER_SESSION,
|
||||
...target, "--platform", options.platform, "--json"], cwd: simulatorProject, environment: controllerEnvironment, timeoutMs: 15 * 60_000 });
|
||||
const shown = async () => {
|
||||
|
||||
@@ -6,6 +6,8 @@ export type CaptureSetupSnapshot = {
|
||||
nodes: readonly Record<string, unknown>[];
|
||||
refsGeneration?: number;
|
||||
truncated?: boolean;
|
||||
/** Whether the controller says a software keyboard shows, when it says (Argent does). */
|
||||
keyboardVisible?: boolean;
|
||||
};
|
||||
|
||||
type Awaitable<T> = T | Promise<T>;
|
||||
@@ -98,13 +100,24 @@ function failure(code: string, message: string, details?: Record<string, unknown
|
||||
...(details ? { details } : {}) });
|
||||
}
|
||||
|
||||
export const SETUP_SELECTOR_MAX = 256;
|
||||
|
||||
/** A label or identifier the executor can select by; a replay plan's selectors are checked against it before any device starts. */
|
||||
export function isSetupSelector(value: unknown): value is string {
|
||||
return typeof value === "string" && value.trim().length > 0 && value.length <= SETUP_SELECTOR_MAX && !/[\u0000-\u001f\u007f]/.test(value);
|
||||
}
|
||||
|
||||
/** Text the executor can fill or type: one line of at most 4,096 characters. */
|
||||
export function isSetupInput(value: unknown): value is string {
|
||||
return typeof value === "string" && value.length <= 4_096 && !/[\u0000\r\n]/.test(value);
|
||||
}
|
||||
|
||||
function selectorValue(selector: CaptureSetupSelector): { key: "label" | "identifier"; value: string } {
|
||||
const hasLabel = typeof selector.label === "string";
|
||||
const hasIdentifier = typeof selector.identifier === "string";
|
||||
const key = hasLabel ? "label" : "identifier";
|
||||
const value = selector[key];
|
||||
if (hasLabel === hasIdentifier || typeof value !== "string" || !value.trim()
|
||||
|| value.length > 256 || /[\u0000-\u001f\u007f]/.test(value)) {
|
||||
if (hasLabel === hasIdentifier || !isSetupSelector(value)) {
|
||||
throw failure("SETUP_SELECTOR_INVALID", "A UI selector must contain exactly one nonempty label or identifier.");
|
||||
}
|
||||
return { key, value };
|
||||
@@ -274,8 +287,7 @@ export async function executeCaptureSetup(actions: readonly CaptureSetupAction[]
|
||||
case "tap":
|
||||
case "fill": {
|
||||
selectorValue(action);
|
||||
if (action.action === "fill" && (typeof action.text !== "string" || action.text.length > 4_096
|
||||
|| /[\u0000\r\n]/.test(action.text))) {
|
||||
if (action.action === "fill" && !isSetupInput(action.text)) {
|
||||
throw failure("SETUP_INPUT_INVALID", "A UI fill input must be a single line of at most 4,096 characters.");
|
||||
}
|
||||
let resolved = false;
|
||||
@@ -330,7 +342,7 @@ export async function executeCaptureSetup(actions: readonly CaptureSetupAction[]
|
||||
break;
|
||||
}
|
||||
case "type": {
|
||||
if (typeof action.text !== "string" || !action.text || action.text.length > 4_096 || /[\u0000\r\n]/.test(action.text)) {
|
||||
if (!isSetupInput(action.text) || !action.text) {
|
||||
throw failure("SETUP_INPUT_INVALID", "A UI type input must be a nonempty single line of at most 4,096 characters.");
|
||||
}
|
||||
let ready = false;
|
||||
|
||||
@@ -1,24 +1,11 @@
|
||||
import { CiError } from "../shared/ci-errors.js";
|
||||
import { isRecord } from "./common.js";
|
||||
|
||||
/** Runs the pinned EAS CLI with these arguments; throws when it exits non-zero unless `allowFailure` is set. */
|
||||
export type EasCommand = (args: string[], options?: { timeoutMs?: number; allowFailure?: boolean }) => { stdout: string; stderr?: string };
|
||||
|
||||
/**
|
||||
* The host actions an EAS Simulator iOS session offers through `eas simulator:control` (serve-sim's actions, 0.10.0 and
|
||||
* later for the ones Explore uses): an empty set when the session or the EAS CLI offers none, so a caller falls back to
|
||||
* what it did before them. The session is always named by id, never taken from a `.env.eas-simulator` beside it.
|
||||
*
|
||||
* These are a fallback: what the session's controller can do itself (agent-device or Argent: fold, appearance, a
|
||||
* reader reset) goes through the controller. Explore uses an action here only for what neither controller offers —
|
||||
* approving a link scheme, restarting SpringBoard, and Argent's appearance (software-mansion/argent#1509).
|
||||
* Runs the pinned EAS CLI with these arguments; throws when it exits non-zero unless `allowFailure` is set. A runner that
|
||||
* knows the exit status reports it as `status`.
|
||||
*/
|
||||
export function simulatorControlActions(eas: EasCommand, sessionId: string): Set<string> {
|
||||
try {
|
||||
const listed = eas(["simulator:control", "--list", "--id", sessionId, "--json", "--non-interactive"], { timeoutMs: 2 * 60_000, allowFailure: true });
|
||||
const actions = JSON.parse(listed.stdout) as unknown;
|
||||
return new Set(isRecord(actions) ? Object.keys(actions) : []);
|
||||
} catch { return new Set(); }
|
||||
}
|
||||
export type EasCommand = (args: string[], options?: { timeoutMs?: number; allowFailure?: boolean }) => { stdout: string; stderr?: string; status?: number | null };
|
||||
|
||||
/**
|
||||
* Pauses before each resend of a failed action. Every action Explore runs is idempotent, and the session's host can fail
|
||||
@@ -27,6 +14,58 @@ export function simulatorControlActions(eas: EasCommand, sessionId: string): Set
|
||||
*/
|
||||
const SIMULATOR_CONTROL_RETRY_DELAYS_MS = [3_000, 6_000] as const;
|
||||
|
||||
/**
|
||||
* What eas-cli 25.2.0 says when the session's serve-sim predates the action list (it refuses `actions.describe`, or
|
||||
* answers it without a catalog), and what an EAS CLI without `simulator:control` says. Those sessions offer no actions,
|
||||
* and asking again would not change that.
|
||||
*/
|
||||
const NO_ACTION_LIST = /serve-sim version that does not support this action|did not return its list of actions|command simulator:control not found/;
|
||||
|
||||
/**
|
||||
* The host actions an EAS Simulator iOS session offers through `eas simulator:control` (serve-sim's actions, 0.10.0 and
|
||||
* later for the ones Explore uses): an empty set when the session's serve-sim or the EAS CLI predates the list, so a
|
||||
* caller falls back to what it did before them. The session is always named by id, never taken from a
|
||||
* `.env.eas-simulator` beside it.
|
||||
*
|
||||
* These are a fallback: what the session's controller can do itself (agent-device or Argent: fold, appearance, a
|
||||
* reader reset) goes through the controller. Explore uses an action here only for what neither controller offers —
|
||||
* approving a link scheme, restarting SpringBoard, and Argent's appearance (software-mansion/argent#1509).
|
||||
*
|
||||
* eas-cli prints the list as JSON on stdout only when it succeeds; a failure prints its reason on stderr. A list that
|
||||
* fails is asked for again after a pause, and one that still fails throws a retryable error rather than reading as "no
|
||||
* actions": in brentvatne/euxy's main baseline run 39487618529, one failed list made the dark-appearance session look as
|
||||
* if it offered none, so it dropped dark, approved no link scheme, and skipped the SpringBoard restart its fold needed,
|
||||
* which failed the target. A session whose host cannot answer three times in a row would fail those actions too, and
|
||||
* capturing on without them publishes a report that silently lacks an appearance; a new session is the way out.
|
||||
*/
|
||||
export function simulatorControlActions(eas: EasCommand, sessionId: string, options: SimulatorControlOptions = {}): Set<string> {
|
||||
const sleep = options.sleep ?? (milliseconds => { Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, milliseconds); });
|
||||
const args = ["simulator:control", "--list", "--id", sessionId, "--json", "--non-interactive"];
|
||||
for (let retry = 0; ; retry += 1) {
|
||||
let failure: string;
|
||||
try {
|
||||
const listed = eas(args, { timeoutMs: 2 * 60_000, allowFailure: true });
|
||||
const actions = (() => { try { return JSON.parse(listed.stdout) as unknown; } catch { return undefined; } })();
|
||||
if ((listed.status ?? 0) === 0 && isRecord(actions)) return new Set(Object.keys(actions));
|
||||
const output = `${listed.stderr ?? ""}\n${listed.stdout}`;
|
||||
if (NO_ACTION_LIST.test(output)) return new Set();
|
||||
// eas-cli ends a failure with "simulator:control command failed."; the line before it says why.
|
||||
failure = output.split("\n").map(line => line.trim()).filter(line => line && !/command failed\.?$/.test(line)).at(-1)?.slice(0, 300) || "it printed no list";
|
||||
} catch (error) {
|
||||
if (NO_ACTION_LIST.test(error instanceof Error ? error.message : String(error))) return new Set();
|
||||
failure = firstLine(error);
|
||||
}
|
||||
const delayMs = SIMULATOR_CONTROL_RETRY_DELAYS_MS[retry];
|
||||
if (delayMs === undefined) throw new CiError({ code: "SIMULATOR_CONTROL_UNAVAILABLE", stage: "session", retryable: true,
|
||||
message: `EAS could not list the simulator:control actions of session ${sessionId} after ${retry + 1} attempts: ${failure}`,
|
||||
nextAction: "Rerun the capture on a new session. Without the list, Explore cannot tell whether the session can switch appearance, approve the app's links, or restart SpringBoard.",
|
||||
details: { sessionId, attempts: retry + 1 } });
|
||||
// A list that succeeds on a later attempt would otherwise leave no trace of the failure in the job log.
|
||||
process.stderr.write(`simulator:control --list failed for session ${sessionId} (${failure}); asking again in ${delayMs / 1000} s\n`);
|
||||
sleep(delayMs);
|
||||
}
|
||||
}
|
||||
|
||||
/** Lets a test replace the pause between resends of a failed action. */
|
||||
export type SimulatorControlOptions = { sleep?: ((milliseconds: number) => void) | undefined };
|
||||
|
||||
|
||||
@@ -15,7 +15,10 @@ import { ARGENT_VERSION } from "../shared/argent-version.js";
|
||||
|
||||
const METADATA = "cache-metadata.json";
|
||||
const CONTRACT = "explore.capture-cache";
|
||||
const MAX_BYTES = 250 * 1024 * 1024;
|
||||
/** The most evidence a capture cache holds; a downloaded baseline is refused past it before it is unpacked. */
|
||||
export const CAPTURE_CACHE_MAX_BYTES = 250 * 1024 * 1024;
|
||||
/** The most files a capture cache holds, besides its metadata. */
|
||||
export const CAPTURE_CACHE_MAX_FILES = 10_000;
|
||||
type Recipe = { inventory: unknown; profile: string; config: string; toolsCommit: string };
|
||||
type FileRecord = { path: string; bytes: number; sha256: string };
|
||||
type CacheMetadata = { contract: typeof CONTRACT; recipe: string; commit: string; files: FileRecord[] };
|
||||
@@ -28,9 +31,9 @@ const recipeKey = ({ inventory, profile, config, toolsCommit }: Recipe) => {
|
||||
|
||||
/** Only settings that can change captured runtime evidence belong in the recipe. */
|
||||
export function captureRecipeConfig(config: ExploreConfig, instructionsSha256?: string): string {
|
||||
// How many sessions capture the evidence (capture.maxSessions) does not change it.
|
||||
const { maxSessions: _maxSessions, ...capture } = config.capture;
|
||||
void _maxSessions;
|
||||
// How many sessions capture the evidence (capture.maxSessions), and how long a review waits for a baseline, do not change it.
|
||||
const { maxSessions: _maxSessions, baselineWaitMinutes: _baselineWaitMinutes, ...capture } = config.capture;
|
||||
void _maxSessions; void _baselineWaitMinutes;
|
||||
return canonicalJson({
|
||||
observer: CAPTURE_OBSERVER_HASH,
|
||||
mode: config.mode,
|
||||
@@ -104,7 +107,7 @@ async function fileInventory(directory: string, path = ""): Promise<FileRecord[]
|
||||
else if (info.isFile()) files.push({ path: child, bytes: info.size, sha256: hash(await readFile(join(directory, child))) });
|
||||
else throw new Error("Capture cache contains an unsupported file");
|
||||
}
|
||||
if (files.length > 10_000 || files.reduce((sum, file) => sum + file.bytes, 0) > MAX_BYTES) {
|
||||
if (files.length > CAPTURE_CACHE_MAX_FILES || files.reduce((sum, file) => sum + file.bytes, 0) > CAPTURE_CACHE_MAX_BYTES) {
|
||||
throw new Error("Capture cache exceeds the evidence size limit");
|
||||
}
|
||||
return files;
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
import { openAsBlob } from "node:fs";
|
||||
import { mkdir, mkdtemp, open, rm, writeFile } from "node:fs/promises";
|
||||
import { dirname, join, relative, resolve } from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { BlobReader, Uint8ArrayWriter, ZipReader, configure } from "@zip.js/zip.js";
|
||||
import { CiError } from "../shared/ci-errors.js";
|
||||
import { silentProgress, type Progress } from "../shared/progress.js";
|
||||
import { retryGitHubLookup } from "../report/github-lookup.js";
|
||||
import { safeRelativePath } from "../binary/archive.js";
|
||||
import { CAPTURE_CACHE_MAX_BYTES, CAPTURE_CACHE_MAX_FILES } from "./artifact-cache.js";
|
||||
import { baselineLookup, githubContext, trustedBaselineRun, type GithubArtifact, type GithubRun } from "./workflow-steps.js";
|
||||
|
||||
/**
|
||||
* A review that waits for main's baseline of its base commit instead of capturing the base itself. Right after a
|
||||
* merge, the next pull request's review starts while the baseline workflow is still capturing the new base: in
|
||||
* brentvatne/euxy's review run 39348956443 it had started 15 s before, so every target captured its base too, about
|
||||
* 10–14 minutes per target instead of 6, on twice the EAS Simulator sessions. A baseline job takes 6–8 minutes and
|
||||
* head capture 5–8, so `runCiReview` starts head at once and waits here only where base would start.
|
||||
*/
|
||||
|
||||
/** `capture.baselineWaitMinutes` when the config sets none: two baseline jobs' worth, so one queued behind another still counts. */
|
||||
export const DEFAULT_BASELINE_WAIT_MINUTES = 15;
|
||||
const POLL_MS = 20_000;
|
||||
/** GitHub's Actions API lists an artifact a little after its upload (#47); a finished job gets this long for its artifact to appear. */
|
||||
const ARTIFACT_LISTING_GRACE_MS = 60_000;
|
||||
/** A zip of the largest capture cache, stored rather than deflated, plus its headers. */
|
||||
const MAX_ARTIFACT_ZIP_BYTES = CAPTURE_CACHE_MAX_BYTES + 16 * 1024 * 1024;
|
||||
|
||||
type GithubJob = { name: string; status?: string; conclusion?: string | null };
|
||||
|
||||
export type BaselineWaitOptions = {
|
||||
/** The baseline run `ci:find-baseline` named; it is checked again here, by the same trust rules. */
|
||||
run: number; sha: string; target: string;
|
||||
/** The longest the review waits, from its first poll (`capture.baselineWaitMinutes`). */
|
||||
timeoutMs: number; pollMs?: number;
|
||||
environment?: NodeJS.ProcessEnv; transport?: typeof fetch; progress?: Progress; signal?: AbortSignal;
|
||||
now?: () => number; sleep?: (milliseconds: number, signal?: AbortSignal) => Promise<void>;
|
||||
};
|
||||
|
||||
/** A failed GitHub read that may succeed later: no answer or a 5xx. A bad token or a missing permission is not. */
|
||||
const transientGitHubFailure = (error: unknown) => error instanceof CiError && error.retryable;
|
||||
const describe = (error: unknown) => error instanceof CiError ? `${error.code}: ${error.message}` : error instanceof Error ? error.message : String(error);
|
||||
const statusText = (status: string | undefined) => (status ?? "queued").replaceAll("_", " ");
|
||||
|
||||
/**
|
||||
* Polls one baseline run until this target's baseline artifact is listed, and answers with it. The run must pass the
|
||||
* trust check a finished baseline passes. Each target's job uploads its own artifact when it finishes, so a target is
|
||||
* reused even when another target's job fails (#48); this target's job failing or being cancelled ends the wait, as do
|
||||
* the time limit, a missing permission and GitHub failures that outlast their retries. The reason says which.
|
||||
*/
|
||||
export async function waitForBaselineArtifact(options: BaselineWaitOptions): Promise<{ artifact: { id: number; run: number } } | { reason: string }> {
|
||||
const now = options.now ?? Date.now;
|
||||
const sleep = options.sleep ?? ((milliseconds: number, signal?: AbortSignal) => delay(milliseconds, undefined, signal ? { signal } : {}));
|
||||
const progress = options.progress ?? silentProgress;
|
||||
const started = now();
|
||||
const short = options.sha.slice(0, 8);
|
||||
try {
|
||||
const lookup = await baselineLookup({ sha: options.sha, target: options.target, ...(options.environment ? { environment: options.environment } : {}),
|
||||
...(options.transport ? { transport: options.transport } : {}),
|
||||
retry: read => retryGitHubLookup(read, transientGitHubFailure, milliseconds => sleep(milliseconds, options.signal)) });
|
||||
if (!lookup) return { reason: "the repository has no baseline workflow" };
|
||||
const { context, get, target } = lookup;
|
||||
// The matrix names each baseline job after its target; an older single-platform workflow's one job is `baseline`.
|
||||
const jobNames = [`baseline (${target})`, ...(target === "ios" ? ["baseline"] : [])];
|
||||
let finishedAt: number | undefined;
|
||||
let logged = "";
|
||||
for (;;) {
|
||||
const run = await get<GithubRun>(`repos/${context.repository}/actions/runs/${options.run}`);
|
||||
if (!trustedBaselineRun(run, lookup)) return { reason: `run ${options.run} is not a baseline run of ${short} on ${lookup.defaultBranch}` };
|
||||
for (const name of lookup.names) {
|
||||
const listed = await get<{ artifacts?: GithubArtifact[] }>(`repos/${context.repository}/actions/runs/${run.id}/artifacts?name=${encodeURIComponent(name)}&per_page=100`);
|
||||
const artifact = (listed?.artifacts ?? []).filter(candidate => !candidate.expired && (candidate.name ?? name) === name
|
||||
&& Number.isSafeInteger(candidate.id) && candidate.id > 0).sort((a, b) => b.id - a.id)[0];
|
||||
if (artifact) return { artifact: { id: artifact.id, run: run.id } };
|
||||
}
|
||||
const jobs = await get<{ jobs?: GithubJob[] }>(`repos/${context.repository}/actions/runs/${run.id}/jobs?per_page=100`);
|
||||
const job = jobs?.jobs?.find(candidate => jobNames.includes(candidate.name));
|
||||
if (job?.status === "completed" && job.conclusion !== "success") {
|
||||
return { reason: `its ${target} job ${job.conclusion === "cancelled" ? "was cancelled" : job.conclusion === "failure" ? "failed" : `ended (${job.conclusion ?? "no conclusion"})`}` };
|
||||
}
|
||||
if (job?.status === "completed" || run.status === "completed") {
|
||||
finishedAt ??= now();
|
||||
if (now() - finishedAt >= ARTIFACT_LISTING_GRACE_MS) return { reason: job ? `its ${target} job finished, but GitHub lists no ${lookup.names[0]} artifact`
|
||||
: `the run ended (${run.conclusion ?? "no conclusion"}) without a ${target} baseline job` };
|
||||
}
|
||||
const waited = now() - started;
|
||||
const state = job ? statusText(job.status) : run.status === "completed" ? "finished" : "not started";
|
||||
if (waited >= options.timeoutMs) return { reason: `its ${target} job is still ${state} after ${Math.round(waited / 60_000)} min` };
|
||||
// A line when the state changes and once a minute, not on every poll.
|
||||
const line = `waiting for ${lookup.defaultBranch}'s baseline run ${run.id} of ${short} (${target}: ${state}, ${Math.floor(waited / 60_000)} min)`;
|
||||
if (line !== logged) { progress(line); logged = line; }
|
||||
await sleep(Math.min(options.pollMs ?? POLL_MS, options.timeoutMs - waited), options.signal);
|
||||
}
|
||||
} catch (error) {
|
||||
return { reason: options.signal?.aborted ? "the review stopped waiting" : describe(error) };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Downloads a baseline artifact and unpacks it into `directory`, which must not exist yet: what
|
||||
* actions/download-artifact would have put in EXPLORE_CAPTURE_CACHE. The token goes to GitHub only, never to the
|
||||
* storage URL GitHub redirects to. Paths are confined to the directory and the cache's size limits apply before
|
||||
* anything is written; restoreCaptureArtifactCache then checks the evidence as it checks a downloaded one.
|
||||
*/
|
||||
export async function downloadBaselineArtifact(options: { artifact: number; directory: string; environment?: NodeJS.ProcessEnv; transport?: typeof fetch;
|
||||
signal?: AbortSignal; sleep?: (milliseconds: number, signal?: AbortSignal) => Promise<void> }): Promise<void> {
|
||||
const environment = options.environment ?? process.env; const transport = options.transport ?? fetch;
|
||||
const context = githubContext(environment);
|
||||
const sleep = options.sleep ?? ((milliseconds: number, signal?: AbortSignal) => delay(milliseconds, undefined, signal ? { signal } : {}));
|
||||
const signal = (milliseconds: number) => options.signal ? AbortSignal.any([options.signal, AbortSignal.timeout(milliseconds)]) : AbortSignal.timeout(milliseconds);
|
||||
const path = `repos/${context.repository}/actions/artifacts/${options.artifact}/zip`;
|
||||
// A 404 just after the artifact was listed is the Actions API catching up (#47), as a 5xx or no answer may be.
|
||||
const located = await retryGitHubLookup(async () => {
|
||||
let response: Response;
|
||||
try {
|
||||
response = await transport(`https://api.github.com/${path}`, { redirect: "manual", signal: signal(60_000),
|
||||
headers: { authorization: `Bearer ${context.token}`, accept: "application/vnd.github+json", "x-github-api-version": "2022-11-28" } });
|
||||
} catch (error) {
|
||||
throw new CiError({ code: "GITHUB_REQUEST_FAILED", stage: "capture-base", message: `GitHub could not be reached (${path}).`,
|
||||
nextAction: "Check GitHub availability.", retryable: true, cause: error });
|
||||
}
|
||||
if (response.status >= 300 && response.status < 400) {
|
||||
await response.body?.cancel().catch(() => {});
|
||||
return { location: response.headers.get("location") ?? "" };
|
||||
}
|
||||
if (response.ok) return { response };
|
||||
await response.body?.cancel().catch(() => {});
|
||||
throw new CiError({ code: response.status === 403 ? "GITHUB_PERMISSION_DENIED" : "GITHUB_REQUEST_FAILED", stage: "capture-base",
|
||||
message: `GitHub answered HTTP ${response.status} for ${path}.`, retryable: response.status === 404 || response.status >= 500,
|
||||
nextAction: response.status === 403 ? "Grant the capture job actions: read." : "Rerun the workflow; check GitHub's status if it keeps failing." });
|
||||
}, transientGitHubFailure, milliseconds => sleep(milliseconds, options.signal));
|
||||
let response = located.response;
|
||||
if (!response) {
|
||||
const location = URL.canParse(located.location ?? "") ? new URL(located.location!) : undefined;
|
||||
if (location?.protocol !== "https:") throw new Error("GitHub redirected the artifact download to no HTTPS address");
|
||||
response = await transport(location.href, { signal: signal(10 * 60_000) });
|
||||
if (!response.ok || !response.body) throw new Error(`The artifact download answered HTTP ${response.status}`);
|
||||
}
|
||||
await mkdir(dirname(resolve(options.directory)), { recursive: true });
|
||||
const stage = await mkdtemp(`${resolve(options.directory)}.download-`);
|
||||
try {
|
||||
const archive = join(stage, "artifact.zip");
|
||||
const file = await open(archive, "wx", 0o600);
|
||||
try {
|
||||
let bytes = 0;
|
||||
for await (const chunk of response.body as unknown as AsyncIterable<Uint8Array>) {
|
||||
bytes += chunk.byteLength;
|
||||
if (bytes > MAX_ARTIFACT_ZIP_BYTES) throw new Error("The baseline artifact is larger than a capture cache can be");
|
||||
await file.write(chunk);
|
||||
}
|
||||
} finally { await file.close(); }
|
||||
await unpackArtifact(archive, options.directory);
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function unpackArtifact(archive: string, directory: string): Promise<void> {
|
||||
configure({ useWebWorkers: false });
|
||||
const zip = new ZipReader(new BlobReader(await openAsBlob(archive)), { useWebWorkers: false });
|
||||
try {
|
||||
const entries = await zip.getEntries();
|
||||
const files = entries.filter(entry => !entry.directory);
|
||||
// The cache's own metadata file is the one file beyond its evidence.
|
||||
if (files.length > CAPTURE_CACHE_MAX_FILES + 1 || files.reduce((sum, entry) => sum + entry.uncompressedSize, 0) > CAPTURE_CACHE_MAX_BYTES + 2 * 1024 * 1024) {
|
||||
throw new Error("The baseline artifact exceeds the capture cache's limits");
|
||||
}
|
||||
const root = resolve(directory);
|
||||
await mkdir(root);
|
||||
for (const entry of files) {
|
||||
const path = safeRelativePath(entry.filename);
|
||||
const target = path ? resolve(root, path) : undefined;
|
||||
if (!target || relative(root, target).startsWith("..") || !entry.getData) throw new Error(`The baseline artifact holds an unsafe path: ${entry.filename}`);
|
||||
await mkdir(dirname(target), { recursive: true });
|
||||
await writeFile(target, await entry.getData(new Uint8ArrayWriter(), { checkSignature: true, useWebWorkers: false }), { flag: "wx", mode: 0o600 });
|
||||
}
|
||||
} finally {
|
||||
await zip.close();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The wait as `capture:review` runs it: the downloaded baseline's directory, or undefined, and then base captures as it
|
||||
* would have. It never fails the review; each outcome is one progress line.
|
||||
*/
|
||||
export async function awaitPendingBaseline(options: BaselineWaitOptions & { directory: string }): Promise<string | undefined> {
|
||||
const progress = options.progress ?? silentProgress;
|
||||
const short = options.sha.slice(0, 8);
|
||||
const outcome = await waitForBaselineArtifact(options);
|
||||
if ("reason" in outcome) {
|
||||
progress(`base: stopped waiting for the baseline of ${short} (${outcome.reason}); capturing the base too`);
|
||||
return undefined;
|
||||
}
|
||||
try {
|
||||
await downloadBaselineArtifact({ ...options, artifact: outcome.artifact.id });
|
||||
progress(`base: downloaded the baseline of ${short} that run ${outcome.artifact.run} captured`);
|
||||
return options.directory;
|
||||
} catch (error) {
|
||||
await rm(options.directory, { recursive: true, force: true }).catch(() => {});
|
||||
progress(`base: the baseline of ${short} from run ${outcome.artifact.run} could not be downloaded (${options.signal?.aborted ? "the review stopped waiting" : describe(error)}); capturing the base too`);
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
+69
-9
@@ -18,6 +18,7 @@ import { quarantineReviewContract } from "../contracts/review-validation.js";
|
||||
import { agentCaptureConstraints } from "../capture/agent/source-context.js";
|
||||
import { captureArtifactCacheMayMatch, captureRecipeConfig, restoreCaptureArtifactCache, writeCaptureArtifactCache } from "./artifact-cache.js";
|
||||
import { runCiMapOnWorker } from "./capture-worker.js";
|
||||
import { DEFAULT_BASELINE_WAIT_MINUTES, awaitPendingBaseline } from "./baseline-wait.js";
|
||||
import { writeCiReportInput, readAppDirectory } from "../report/artifact.js";
|
||||
import { assessPublicEvidence, readExportedMap, type PublicEvidenceSide } from "../review/evidence-assessment.js";
|
||||
import { parseRuntimeExecutionContext } from "../review/runtime-adapter.js";
|
||||
@@ -173,6 +174,24 @@ export async function sideAppearancePasses(config: ExploreConfig, environment: N
|
||||
return passes.length > 1 ? passes.map(pass => [...pass]) : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Base's start in a review that waits for a baseline still being captured; head has already started. Base captures as
|
||||
* soon as the wait gives up, beside head when the sides capture concurrently, and a downloaded baseline is restored
|
||||
* once head finishes, since restoring checks it against head's runtime profile. A head that fails ends the wait, and
|
||||
* base is never started for a review that has already failed.
|
||||
*/
|
||||
export async function captureBaseAfterWait(input: { waited: Promise<string | undefined> | undefined; head: Promise<void>; stopWaiting: () => void;
|
||||
captureBase: (cacheDirectory?: string) => Promise<void> }): Promise<void> {
|
||||
if (!input.waited) return input.captureBase();
|
||||
let headFailed = false;
|
||||
input.head.catch(() => { headFailed = true; input.stopWaiting(); });
|
||||
const waited = await input.waited;
|
||||
if (headFailed) return;
|
||||
if (!waited) return input.captureBase();
|
||||
await input.head;
|
||||
return input.captureBase(waited);
|
||||
}
|
||||
|
||||
export function captureSidesConcurrently(input: { plan: SourceCapturePlan; agent: boolean; injectedRunner: boolean; baselineMayExist: boolean }): boolean {
|
||||
return !input.agent && !input.injectedRunner && !input.baselineMayExist
|
||||
&& input.plan.headCaptureSurfaceIds.length > 0 && input.plan.baseCaptureSurfaceIds.length > 0;
|
||||
@@ -208,9 +227,30 @@ export type PublicCiOptions = {
|
||||
runner?: CommandRunner;
|
||||
/** Internal dependency injection; configuration cannot replace the authenticated CLI session. */
|
||||
agentSessionFactory?: PrepareCiExplorationOptions["createSession"];
|
||||
/** Internal dependency injection for tests: replaces the wait for a baseline run that EXPLORE_BASELINE_RUN names. */
|
||||
awaitBaseline?: BaselineWait;
|
||||
progress?: Progress;
|
||||
};
|
||||
|
||||
/** Waits for a baseline of the base commit still being captured: its downloaded export's directory, or undefined. */
|
||||
type BaselineWait = (signal: AbortSignal) => Promise<string | undefined>;
|
||||
|
||||
/**
|
||||
* The wait for a baseline still capturing the base commit: for the run `ci:find-baseline` named in
|
||||
* EXPLORE_BASELINE_RUN, which the wait checks again by the same trust rules, for up to capture.baselineWaitMinutes. A
|
||||
* workflow that names no run, or a config that sets 0, captures the base as before.
|
||||
*/
|
||||
function pendingBaselineWait(options: Pick<PublicCiOptions, "awaitBaseline" | "config" | "target" | "platform" | "output" | "progress">,
|
||||
environment: NodeJS.ProcessEnv, transport: typeof fetch, baseSha: string): BaselineWait | undefined {
|
||||
if (options.awaitBaseline) return options.awaitBaseline;
|
||||
const run = Number(environment.EXPLORE_BASELINE_RUN);
|
||||
const minutes = options.config.capture.baselineWaitMinutes ?? DEFAULT_BASELINE_WAIT_MINUTES;
|
||||
if (!Number.isSafeInteger(run) || run <= 0 || minutes === 0) return undefined;
|
||||
const target = options.target ?? options.platform ?? options.config.target?.id ?? "ios";
|
||||
return signal => awaitPendingBaseline({ run, sha: baseSha, target, timeoutMs: minutes * 60_000, directory: join(resolve(options.output), "baseline-download"),
|
||||
environment, transport, signal, ...(options.progress ? { progress: options.progress } : {}) });
|
||||
}
|
||||
|
||||
function assertRunIdentity(environment: NodeJS.ProcessEnv): void {
|
||||
if (!/^[1-9][0-9]*$/.test(environment.GITHUB_RUN_ID ?? "") || !/^[1-9][0-9]*$/.test(environment.GITHUB_RUN_ATTEMPT ?? "")) {
|
||||
throw new CiError({ code: "CI_RUN_IDENTITY_MISSING", stage: "preflight", message: "The GitHub run ID or attempt is missing.",
|
||||
@@ -247,13 +287,14 @@ async function prepareCiRun(options: Pick<PublicCiOptions, "baseProject" | "head
|
||||
}
|
||||
|
||||
/** A review of any two commits (`capture:review --compare FILE`): the same capture as a PR's, with no PR to verify or comment on. */
|
||||
export async function runCompareCi(options: Omit<PublicCiOptions, "pullRequestPath" | "transport" | "onComment"> & { comparePath: string }) {
|
||||
export async function runCompareCi(options: Omit<PublicCiOptions, "pullRequestPath" | "onComment"> & { comparePath: string }) {
|
||||
const environment = options.environment ?? process.env;
|
||||
assertRunIdentity(environment);
|
||||
const comparison = projectComparePresentation(JSON.parse(await readFile(options.comparePath, "utf8")));
|
||||
const { projects, output, state } = await prepareCiRun(options, environment, comparison);
|
||||
const assertCurrentHead = async () => { for (const side of ["base", "head"] as const) assertCheckout(projects[side], comparison[side].sha); };
|
||||
return runCiReview({ ...options, projects, output, state, presentation: comparison,
|
||||
const awaitBaseline = pendingBaselineWait(options, environment, options.transport ?? fetch, comparison.base.sha);
|
||||
return runCiReview({ ...options, projects, output, state, presentation: comparison, ...(awaitBaseline ? { awaitBaseline } : {}),
|
||||
environment, runId: environment.GITHUB_RUN_ID!, runAttempt: environment.GITHUB_RUN_ATTEMPT!,
|
||||
assertCurrentHead, postComment: async () => {} });
|
||||
}
|
||||
@@ -324,7 +365,8 @@ export async function runPublicCi(options: PublicCiOptions) {
|
||||
for (const side of ["base", "head"] as const) assertCheckout(projects[side], pr[side].sha);
|
||||
await assertCurrentPr();
|
||||
};
|
||||
return runCiReview({ ...options, projects, output, state, presentation: pr,
|
||||
const awaitBaseline = pendingBaselineWait(options, environment, transport, pr.base.sha);
|
||||
return runCiReview({ ...options, projects, output, state, presentation: pr, ...(awaitBaseline ? { awaitBaseline } : {}),
|
||||
environment, runId: environment.GITHUB_RUN_ID!, runAttempt: environment.GITHUB_RUN_ATTEMPT!,
|
||||
assertCurrentHead, postComment, canPublish: () => !prVerificationFailed });
|
||||
}
|
||||
@@ -354,6 +396,11 @@ export async function runCiReview(options: {
|
||||
now?: () => number; sleep?: (milliseconds: number) => void; runner?: CommandRunner; subscription?: boolean; useLocalEasSession?: boolean;
|
||||
agentSessionFactory?: PrepareCiExplorationOptions["createSession"];
|
||||
prepareCapture?: () => Promise<void>;
|
||||
/**
|
||||
* Waits for a baseline of the base commit that is still being captured, when no downloaded baseline matches. It runs
|
||||
* alongside head, and base waits for it only where base would start capturing.
|
||||
*/
|
||||
awaitBaseline?: BaselineWait;
|
||||
progress?: Progress;
|
||||
/** Stops the captures between steps; model requests take it through agentSessionFactory. */
|
||||
signal?: AbortSignal;
|
||||
@@ -371,6 +418,8 @@ export async function runCiReview(options: {
|
||||
let usagePhase: AgentUsagePhase = "preflight";
|
||||
let preserveReport: (() => Promise<void>) | undefined;
|
||||
let reportWritten = false;
|
||||
// Stops a baseline wait still polling once the review no longer needs it: done, failed, or stopped.
|
||||
const stopWaiting = new AbortController();
|
||||
try {
|
||||
progress(`planning the review of ${pr.head.sha.slice(0, 8)} against base ${pr.base.sha.slice(0, 8)} (scope ${config.exploration.scope})`);
|
||||
const plan = await buildVerifiedPlan({ baseProject: projects.base, headProject: projects.head,
|
||||
@@ -401,6 +450,10 @@ export async function runCiReview(options: {
|
||||
// A baseline that may be restored needs the head runtime profile first; otherwise both sides capture at once.
|
||||
const baselineMayExist = plan.baseCaptureSurfaceIds.length > 0 && await captureArtifactCacheMayMatch({ ...cacheRecipes.base,
|
||||
...(environment.EXPLORE_CAPTURE_CACHE ? { sourceDirectory: environment.EXPLORE_CAPTURE_CACHE } : {}), commit: pr.base.sha });
|
||||
// With no finished baseline to reuse, one still being captured is waited for: from now, alongside head, and base
|
||||
// waits for it only where it would start. Head is never delayed by it (brentvatne/euxy's review run 39348956443).
|
||||
const waitedBaseline = !baselineMayExist && plan.baseCaptureSurfaceIds.length > 0 && options.awaitBaseline
|
||||
? options.awaitBaseline(options.signal ? AbortSignal.any([stopWaiting.signal, options.signal]) : stopWaiting.signal).catch(() => undefined) : undefined;
|
||||
const share = targetSessionShare(config, options.targets, options.posedTargets);
|
||||
// A share of one session captures the sides in turn.
|
||||
const concurrent = captureSidesConcurrently({ plan, agent: !!exploration.agent, injectedRunner: !!options.runner, baselineMayExist }) && share >= 2;
|
||||
@@ -415,13 +468,13 @@ export async function runCiReview(options: {
|
||||
progress(`sessions: ${share} of ${config.capture.maxSessions ?? DEFAULT_MAX_SESSIONS} for this target; `
|
||||
+ (split.length ? `${split.join(" and ")} ${split.length === 1 ? "captures its" : "capture their"} appearances in separate sessions`
|
||||
: concurrent ? "one session per side" : "the sides in turn"));
|
||||
const captureSide = async (side: "head" | "base") => {
|
||||
const captureSide = async (side: "head" | "base", cacheDirectory = environment.EXPLORE_CAPTURE_CACHE) => {
|
||||
const directory = join(output, `${side}-export`);
|
||||
const selectedIds = plan[`${side}CaptureSurfaceIds`];
|
||||
const cacheRecipe = cacheRecipes[side];
|
||||
let reuseDirectory: string | undefined;
|
||||
if (side === "base" && selectedIds.length && runtimeProfile && environment.EXPLORE_CAPTURE_CACHE) {
|
||||
const restored = await restoreCaptureArtifactCache({ ...cacheRecipe, sourceDirectory: environment.EXPLORE_CAPTURE_CACHE,
|
||||
if (side === "base" && selectedIds.length && runtimeProfile && cacheDirectory) {
|
||||
const restored = await restoreCaptureArtifactCache({ ...cacheRecipe, sourceDirectory: cacheDirectory,
|
||||
commit: pr.base.sha, selectedIds, directory: join(output, "base-cache"), runtimeProfile,
|
||||
...(exploration.agent ? { captureContexts: exploration.agent.contexts } : {}) });
|
||||
if (restored) reuseDirectory = join(output, "base-cache");
|
||||
@@ -453,12 +506,17 @@ export async function runCiReview(options: {
|
||||
}
|
||||
await writeCaptureArtifactCache(directory, cacheRecipe);
|
||||
};
|
||||
const captureBase = (head: Promise<void>) => captureBaseAfterWait({ waited: waitedBaseline, head, stopWaiting: () => stopWaiting.abort(),
|
||||
captureBase: cacheDirectory => captureSide("base", cacheDirectory) });
|
||||
if (waitedBaseline) progress(`base waits for the baseline run of ${pr.base.sha.slice(0, 8)} still being captured; head captures meanwhile`);
|
||||
if (concurrent) {
|
||||
stage = "capture-head"; state.phase = "capturing"; state.capture.current = "head"; usagePhase = "capture-head";
|
||||
progress("capturing head and base at the same time in two simulator sessions");
|
||||
progress(waitedBaseline ? "capturing head; base captures in a second simulator session if the baseline cannot be reused"
|
||||
: "capturing head and base at the same time in two simulator sessions");
|
||||
await postComment();
|
||||
// Let both sides finish or fail on their own so every started session is stopped by its owner.
|
||||
const settled = await Promise.allSettled([captureSide("head"), captureSide("base")]);
|
||||
const head = captureSide("head");
|
||||
const settled = await Promise.allSettled([head, captureBase(head)]);
|
||||
for (const [index, outcome] of settled.entries()) {
|
||||
if (outcome.status === "rejected") { stage = index === 0 ? "capture-head" : "capture-base"; throw outcome.reason; }
|
||||
}
|
||||
@@ -467,7 +525,7 @@ export async function runCiReview(options: {
|
||||
stage = `capture-${side}`; state.phase = "capturing"; state.capture.current = side;
|
||||
usagePhase = `capture-${side}`;
|
||||
await postComment();
|
||||
await captureSide(side);
|
||||
await (side === "head" ? captureSide("head") : captureBase(Promise.resolve()));
|
||||
}
|
||||
}
|
||||
delete state.capture.current;
|
||||
@@ -564,6 +622,8 @@ export async function runCiReview(options: {
|
||||
try { await postComment(); } catch { /* A stale PR or inaccessible GitHub must not replace the original failure. */ }
|
||||
}
|
||||
throw error;
|
||||
} finally {
|
||||
stopWaiting.abort();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+70
-21
@@ -30,7 +30,7 @@ export async function writeCaptureOutputs(inputPath: string, environment: Enviro
|
||||
await writeStepOutputs(environment, { preview: (await readCiReportInput(inputPath)).preview?.hosting ?? "" });
|
||||
}
|
||||
|
||||
function githubContext(environment: Environment) {
|
||||
export function githubContext(environment: Environment) {
|
||||
const repository = environment.GITHUB_REPOSITORY ?? "";
|
||||
const token = environment.GITHUB_TOKEN?.trim();
|
||||
if (!/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/.test(repository)) throw new CiError({ code: "CI_RUN_IDENTITY_MISSING", stage: "preflight",
|
||||
@@ -41,7 +41,7 @@ function githubContext(environment: Environment) {
|
||||
}
|
||||
|
||||
/** One GitHub REST read; 404 is an answer (undefined), every other failure stops the step with what to do. */
|
||||
async function githubGet<T>(path: string, context: { token: string }, transport: typeof fetch): Promise<T | undefined> {
|
||||
export async function githubGet<T>(path: string, context: { token: string }, transport: typeof fetch): Promise<T | undefined> {
|
||||
let response: Response;
|
||||
try {
|
||||
response = await transport(`https://api.github.com/${path}`, { redirect: "error", signal: AbortSignal.timeout(30_000),
|
||||
@@ -102,60 +102,109 @@ async function trustedAuthors(context: { repository: string; token: string }, tr
|
||||
catch (error) { return { logins: [], invalid: error instanceof Error ? error.message : String(error) }; }
|
||||
}
|
||||
|
||||
type GithubArtifact = { id: number; expired: boolean; workflow_run?: { id?: number } };
|
||||
type GithubRun = { conclusion: string | null; workflow_id: number; head_sha: string; head_branch: string; event: string; head_repository?: { full_name?: string } };
|
||||
export type GithubArtifact = { id: number; name?: string; expired: boolean; workflow_run?: { id?: number } };
|
||||
export type GithubRun = { id: number; status?: string; conclusion: string | null; workflow_id: number; head_sha: string; head_branch: string; event: string;
|
||||
head_repository?: { full_name?: string } };
|
||||
|
||||
/**
|
||||
* `ci:find-baseline`: the newest baseline-workflow artifact of exactly this commit, uploaded from the default branch of
|
||||
* this repository by a finished run. Its id and run go to the step outputs for actions/download-artifact. A run that
|
||||
* failed counts: a target's baseline job uploads its artifact only after every step before it passed, so another
|
||||
* target's failure (one flaky simulator session, as in brentvatne/euxy#172) does not make this target's baseline
|
||||
* unusable. A run still in progress or cancelled does not.
|
||||
* What every baseline lookup starts from: the GitHub context, a checked commit and target, the default branch and the
|
||||
* baseline workflow's id, and the artifact names a baseline of this target may carry. Undefined when the repository
|
||||
* has no baseline workflow.
|
||||
*/
|
||||
export async function findBaselineArtifact(options: { sha: string; target?: string;
|
||||
/** What callers from before targets pass: a platform is the id of its default target. */ platform?: string; environment?: Environment; transport?: typeof fetch }): Promise<{ id: number; run: number } | undefined> {
|
||||
export async function baselineLookup(options: { sha: string; target?: string; platform?: string; environment?: Environment; transport?: typeof fetch;
|
||||
/** Wraps each read, as a wait that polls for minutes retries GitHub's transient failures. */ retry?: <T>(read: () => Promise<T>) => Promise<T> }) {
|
||||
const environment = options.environment ?? process.env; const transport = options.transport ?? fetch;
|
||||
const context = githubContext(environment);
|
||||
if (!SHA.test(options.sha)) throw new CiError({ code: "SHA_INVALID", stage: "preflight", message: "The base commit must be a full SHA.", nextAction: "Pass --sha with a 40-character commit SHA." });
|
||||
const target = options.target ?? options.platform ?? "ios";
|
||||
if (!isTargetId(target)) throw new CiError({ code: "TARGET_INVALID", stage: "preflight", message: `${target} is not a target id.`,
|
||||
nextAction: "Pass --target with an id from targets: in .expo-explore/config.yaml (ios or android without targets:)." });
|
||||
const repository = await githubGet<{ default_branch: string }>(`repos/${context.repository}`, context, transport);
|
||||
const workflow = await githubGet<{ id: number }>(`repos/${context.repository}/actions/workflows/explore-baseline.yml`, context, transport);
|
||||
const read = <T>(path: string) => githubGet<T>(path, context, transport);
|
||||
const get = <T>(path: string) => options.retry ? options.retry(() => read<T>(path)) : read<T>(path);
|
||||
const repository = await get<{ default_branch: string }>(`repos/${context.repository}`);
|
||||
const workflow = await get<{ id: number }>(`repos/${context.repository}/actions/workflows/explore-baseline.yml`);
|
||||
if (!repository || !workflow) return undefined;
|
||||
// Baselines carry the target suffix; target ios also accepts the unsuffixed name older single-platform baseline workflows uploaded.
|
||||
const names = [`explore-baseline-${options.sha}-${target}`, ...(target === "ios" ? [`explore-baseline-${options.sha}`] : [])];
|
||||
return { context, get, target, sha: options.sha, defaultBranch: repository.default_branch, workflow: workflow.id, names };
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a run may supply this commit's baseline: a run of the baseline workflow for exactly this commit, on the
|
||||
* default branch of this repository, started by a push or a manual dispatch. A finished run's artifact is reused, and
|
||||
* a run still capturing is waited for, only when it passes this check; nothing from a pull request's head selects it.
|
||||
*/
|
||||
export function trustedBaselineRun(run: GithubRun | undefined, lookup: { workflow: number; sha: string; defaultBranch: string; context: { repository: string } }): run is GithubRun {
|
||||
return !!run && run.workflow_id === lookup.workflow && run.head_sha === lookup.sha && run.head_branch === lookup.defaultBranch
|
||||
&& ["push", "workflow_dispatch"].includes(run.event) && run.head_repository?.full_name === lookup.context.repository;
|
||||
}
|
||||
|
||||
/**
|
||||
* `ci:find-baseline`: the newest baseline-workflow artifact of exactly this commit, uploaded from the default branch of
|
||||
* this repository by a finished run. Its id and run go to the step outputs for actions/download-artifact. A run that
|
||||
* failed counts: a target's baseline job uploads its artifact only after every step before it passed, so another
|
||||
* target's failure (one flaky simulator session, as in brentvatne/euxy#172) does not make this target's baseline
|
||||
* unusable. A run still in progress or cancelled does not; findPendingBaselineRun finds one still capturing.
|
||||
*/
|
||||
export async function findBaselineArtifact(options: { sha: string; target?: string;
|
||||
/** What callers from before targets pass: a platform is the id of its default target. */ platform?: string; environment?: Environment; transport?: typeof fetch }): Promise<{ id: number; run: number } | undefined> {
|
||||
const lookup = await baselineLookup(options);
|
||||
if (!lookup) return undefined;
|
||||
const { context, get } = lookup;
|
||||
const artifacts: GithubArtifact[] = [];
|
||||
for (const name of names) for (let page = 1; page <= 10; page++) {
|
||||
const listed = await githubGet<{ artifacts: GithubArtifact[] }>(`repos/${context.repository}/actions/artifacts?name=${encodeURIComponent(name)}&per_page=100&page=${page}`, context, transport);
|
||||
for (const name of lookup.names) for (let page = 1; page <= 10; page++) {
|
||||
const listed = await get<{ artifacts: GithubArtifact[] }>(`repos/${context.repository}/actions/artifacts?name=${encodeURIComponent(name)}&per_page=100&page=${page}`);
|
||||
artifacts.push(...listed?.artifacts ?? []);
|
||||
if ((listed?.artifacts.length ?? 0) < 100) break;
|
||||
}
|
||||
for (const artifact of artifacts.filter(candidate => !candidate.expired).sort((a, b) => b.id - a.id)) {
|
||||
if (!artifact.workflow_run?.id) continue;
|
||||
const run = await githubGet<GithubRun>(`repos/${context.repository}/actions/runs/${artifact.workflow_run.id}`, context, transport);
|
||||
if (!run || (run.conclusion !== "success" && run.conclusion !== "failure") || run.workflow_id !== workflow.id || run.head_sha !== options.sha
|
||||
|| run.head_branch !== repository.default_branch || !["push", "workflow_dispatch"].includes(run.event)
|
||||
|| run.head_repository?.full_name !== context.repository) continue;
|
||||
const run = await get<GithubRun>(`repos/${context.repository}/actions/runs/${artifact.workflow_run.id}`);
|
||||
if (!trustedBaselineRun(run, lookup) || (run.conclusion !== "success" && run.conclusion !== "failure")) continue;
|
||||
return { id: artifact.id, run: artifact.workflow_run.id };
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* A baseline run of exactly this commit that has not finished yet (queued, waiting or in progress), passing the same
|
||||
* trust check as a finished one. In brentvatne/euxy's review run 39348956443 main's baseline of the base commit had
|
||||
* started 15 s before the review, so no artifact existed yet. A run already capturing is preferred to one queued
|
||||
* behind it in the baseline's concurrency group, since it finishes first; then the newest.
|
||||
*/
|
||||
export async function findPendingBaselineRun(options: Parameters<typeof baselineLookup>[0]): Promise<{ run: number } | undefined> {
|
||||
const lookup = await baselineLookup(options);
|
||||
if (!lookup) return undefined;
|
||||
const listed = await lookup.get<{ workflow_runs?: GithubRun[] }>(`repos/${lookup.context.repository}/actions/workflows/${lookup.workflow}/runs?head_sha=${lookup.sha}`
|
||||
+ `&branch=${encodeURIComponent(lookup.defaultBranch)}&per_page=100`);
|
||||
const pending = (listed?.workflow_runs ?? []).filter(run => run.status !== "completed" && Number.isSafeInteger(run.id) && run.id > 0 && trustedBaselineRun(run, lookup))
|
||||
.sort((a, b) => Number(b.status === "in_progress") - Number(a.status === "in_progress") || b.id - a.id);
|
||||
return pending[0] ? { run: pending[0].id } : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* `ci:find-baseline` as the workflow runs it: a baseline is optional, so a lookup that fails (a GitHub 5xx, a rate
|
||||
* limit) is a warning and no baseline, never an error. A `ci:*` error would write explore-diagnostics/error.json,
|
||||
* which the failure comment would then report as the cause of an unrelated failure, possibly on another target.
|
||||
* With no finished baseline, it names a trusted baseline run still capturing the commit as `pending`, which
|
||||
* `capture:review` waits for alongside head.
|
||||
*/
|
||||
export async function findOptionalBaseline(options: Parameters<typeof findBaselineArtifact>[0]):
|
||||
Promise<{ baseline?: { id: number; run: number }; warning?: { code: string; message: string } }> {
|
||||
Promise<{ baseline?: { id: number; run: number }; pending?: { run: number }; warning?: { code: string; message: string } }> {
|
||||
try {
|
||||
const baseline = await findBaselineArtifact(options);
|
||||
return baseline ? { baseline } : {};
|
||||
if (baseline) return { baseline };
|
||||
} catch (error) {
|
||||
return { warning: { code: error instanceof CiError ? error.code : "BASELINE_LOOKUP_FAILED",
|
||||
message: `No baseline is reused, so capture captures the base too: ${error instanceof Error ? error.message : String(error)}` } };
|
||||
}
|
||||
try {
|
||||
const pending = await findPendingBaselineRun(options);
|
||||
return pending ? { pending } : {};
|
||||
} catch (error) {
|
||||
return { warning: { code: error instanceof CiError ? error.code : "BASELINE_LOOKUP_FAILED",
|
||||
message: `No baseline run in progress is waited for, so capture captures the base too: ${error instanceof Error ? error.message : String(error)}` } };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+2
-2
@@ -90,10 +90,10 @@ const flagKinds: Record<string, Record<string, "value" | "boolean">> = {
|
||||
|
||||
|
||||
|
||||
"explore:binary": { archive:"value", output:"value", platform:"value", controller:"value", simulator:"value", device:"value", "bundle-id":"value", "launch-url":"value", hints:"value", "allow-destructive":"value",
|
||||
"explore:binary": { replay:"value", archive:"value", output:"value", platform:"value", controller:"value", simulator:"value", device:"value", "bundle-id":"value", "launch-url":"value", hints:"value", "allow-destructive":"value",
|
||||
"max-duration-minutes":"value", "max-screens":"value", "max-actions":"value", provider:"value", model:"value", owner:"value", project:"value",
|
||||
upload:"boolean", "no-upload":"boolean", "expires-in":"value", alias:"value", "no-open":"boolean", port:"value" },
|
||||
"explore:session": { session:"value", "env-file":"value", "bundle-id":"value", "app-name":"value", "app-version":"value", output:"value", platform:"value", controller:"value", device:"value", "launch-url":"value", hints:"value",
|
||||
"explore:session": { replay:"value", session:"value", "env-file":"value", "bundle-id":"value", "app-name":"value", "app-version":"value", output:"value", platform:"value", controller:"value", device:"value", "launch-url":"value", hints:"value",
|
||||
"allow-destructive":"value", "max-duration-minutes":"value", "max-screens":"value", "max-actions":"value", provider:"value", model:"value", owner:"value", project:"value",
|
||||
upload:"boolean", "expires-in":"value", alias:"value", "no-open":"boolean", port:"value" },
|
||||
"ci:report": { input:"value", output:"value", target:"value", platform:"value" },
|
||||
|
||||
@@ -7,6 +7,7 @@ import { assertHostingExport, deployExplorationReport, zipReportDirectory } from
|
||||
import { keepReports } from "./report-commands.js";
|
||||
import { warnIgnoredExpiry } from "../config/explore-config.js";
|
||||
import { argumentError, flag, flags, has, nonNegativeInteger, oneOf, output, positional, positiveInteger } from "./args.js";
|
||||
import { readReplayPlan } from "../binary/replay.js";
|
||||
|
||||
|
||||
/** `--alias NAME` plus a fixed token from EXPLORE_PREVIEW_TOKEN: the hosted link stays the same from one run to the next (validated by the deploy). */
|
||||
@@ -49,6 +50,7 @@ export async function exploreBinary(): Promise<void> {
|
||||
...(flag("max-duration-minutes") ? { maxDurationMinutes: positiveInteger("max-duration-minutes") } : {}),
|
||||
...(flag("max-screens") ? { maxScreens: positiveInteger("max-screens") } : {}),
|
||||
...(flag("max-actions") ? { maxActions: positiveInteger("max-actions") } : {}),
|
||||
...(flag("replay") ? { replay: await readReplayPlan(resolve(process.cwd(), flag("replay")!)) } : {}),
|
||||
});
|
||||
say(`exploration ${result.exploration.stopReason}: ${result.exploration.screens} screens, ${result.exploration.observations} captures, ${result.exploration.actions} actions`);
|
||||
const report = await writeExplorationReport(result, join(outputDirectory, "report"));
|
||||
@@ -113,6 +115,7 @@ export async function exploreSession(): Promise<void> {
|
||||
...(flag("max-duration-minutes") ? { maxDurationMinutes: positiveInteger("max-duration-minutes") } : {}),
|
||||
...(flag("max-screens") ? { maxScreens: positiveInteger("max-screens") } : {}),
|
||||
...(flag("max-actions") ? { maxActions: positiveInteger("max-actions") } : {}),
|
||||
...(flag("replay") ? { replay: await readReplayPlan(resolve(process.cwd(), flag("replay")!)) } : {}),
|
||||
});
|
||||
say(`exploration ${result.exploration.stopReason}: ${result.exploration.screens} screens, ${result.exploration.observations} captures, ${result.exploration.actions} actions`);
|
||||
const report = await writeExplorationReport(result, join(outputDirectory, "report"));
|
||||
|
||||
+10
-3
@@ -110,13 +110,20 @@ export async function ciPullRequest(): Promise<void> {
|
||||
output({ ok: true, pullRequest: presentation, author });
|
||||
}
|
||||
|
||||
/** `ci:find-baseline --sha SHA [--platform P]`: a reusable baseline artifact of the base commit, if one exists. */
|
||||
/**
|
||||
* `ci:find-baseline --sha SHA [--platform P]`: a reusable baseline artifact of the base commit, if one exists, or else
|
||||
* the baseline run still capturing it (`pending`), which `capture:review` waits for alongside head.
|
||||
*/
|
||||
export async function ciFindBaseline(): Promise<void> {
|
||||
const platform = platformFlag(), target = targetFlag();
|
||||
const { baseline, warning } = await findOptionalBaseline({ sha: required("sha"), ...(platform ? { platform } : {}), ...(target ? { target } : {}) });
|
||||
const { baseline, pending, warning } = await findOptionalBaseline({ sha: required("sha"), ...(platform ? { platform } : {}), ...(target ? { target } : {}) });
|
||||
if (baseline) await writeStepOutputs(process.env, { id: String(baseline.id), run: String(baseline.run) });
|
||||
if (pending) {
|
||||
await writeStepOutputs(process.env, { pending: String(pending.run) });
|
||||
console.error(`No finished baseline of ${required("sha").slice(0, 8)} yet; baseline run ${pending.run} is still capturing it; capture:review waits for it while head captures, up to capture.baselineWaitMinutes.`);
|
||||
}
|
||||
if (warning && process.env.GITHUB_ACTIONS) console.log(`::warning title=${warning.code}::${warning.message.replace(/[\r\n]+/g, " ")}`);
|
||||
output({ ok: true, baseline: baseline ?? null, ...(warning ? { warning } : {}) });
|
||||
output({ ok: true, baseline: baseline ?? null, ...(pending ? { pending } : {}), ...(warning ? { warning } : {}) });
|
||||
}
|
||||
|
||||
/** `ci:compare-refs --project DIR --base REF --head REF [--output FILE]`: the two commits a comparison captures. */
|
||||
|
||||
@@ -56,6 +56,7 @@ const BINARY = `Prebuilt apps (no source needed; runs with your local claude or
|
||||
and the model and the report only see the reference)
|
||||
--max-duration-minutes N --max-screens N --max-actions N --provider anthropic|openai --model ID
|
||||
--owner ACCOUNT --project SLUG --allow-destructive "Label,Other label"
|
||||
--replay FILE (takes the steps in FILE instead of a model's, so every run and device takes the same path; no model access needed, and not with --provider, --model or --hints)
|
||||
--alias NAME (one EAS Hosting host per app that always serves the newest run; with EXPLORE_PREVIEW_TOKEN
|
||||
set to a fixed 32-character token, the hosted link is the same every run)
|
||||
(iOS needs a simulator build; a device .ipa and Android .aab are rejected before any device starts)
|
||||
@@ -94,6 +95,7 @@ const CI = `CI (the installed workflows run these; they manage remote EAS Simula
|
||||
(resolves the live PR and refuses it unless it is open, from this repository, and by a trusted author)
|
||||
ci:compare-refs --project DIRECTORY --base REF --head REF [--output FILE]
|
||||
ci:find-baseline --sha COMMIT [--target ID | --platform ios|android]
|
||||
(a finished baseline's artifact of the commit, or else the default branch's baseline run still capturing it)
|
||||
ci:after-capture --output TEMP_DIRECTORY --checkout DIRECTORY...
|
||||
(stops the capture's EAS Simulator sessions and checks capture left each checkout unchanged)
|
||||
(the ci:* commands write their results to $GITHUB_OUTPUT for the next steps)`;
|
||||
|
||||
@@ -99,6 +99,11 @@ export type ExploreConfig = {
|
||||
* DEFAULT_MAX_SESSIONS (5). Below the number of targets, that many target jobs run at once, one session each.
|
||||
*/
|
||||
maxSessions?: number;
|
||||
/**
|
||||
* How many minutes a review waits for the default branch's baseline run of its base commit, still capturing when
|
||||
* the review starts, before it captures the base itself; 0 never waits. Absent: DEFAULT_BASELINE_WAIT_MINUTES (15).
|
||||
*/
|
||||
baselineWaitMinutes?: number;
|
||||
/** Where the route observer is bundled into the app, when the config chooses; captureRepack gives the default. */
|
||||
repack?: "eas" | "local"; appearance: AppearanceSetting; launch: LaunchSetting; reset: RouteResetSetting; unattributed: UnattributedSetting; readinessSeconds?: number;
|
||||
/** The device controller; absent means agent-device, so configs that never chose one keep their cache recipes. */
|
||||
@@ -194,6 +199,11 @@ function positiveInteger(value: unknown, label: string, fallback?: number): numb
|
||||
return value;
|
||||
}
|
||||
|
||||
function boundedInteger(value: unknown, label: string, minimum: number, maximum: number): number {
|
||||
if (typeof value !== "number" || !Number.isSafeInteger(value) || value < minimum || value > maximum) fail(`${label} must be a whole number from ${minimum} to ${maximum}`, label);
|
||||
return value;
|
||||
}
|
||||
|
||||
function customScheme(value: unknown, label: string): string {
|
||||
const result = text(value, label, 128).toLowerCase();
|
||||
if (!/^[a-z][a-z0-9+.-]*$/.test(result) || RESERVED_SCHEMES.has(result)) {
|
||||
@@ -300,7 +310,7 @@ export function validateExploreConfig(value: unknown): ExploreConfig {
|
||||
const exploration = optionalRecord(root.exploration, "exploration");
|
||||
exactKeys(exploration, ["scope", "exclude"], "exploration");
|
||||
const capture = optionalRecord(root.capture, "capture");
|
||||
exactKeys(capture, ["maxMinutes", "maxActions", "maxSessions", "repack", "appearance", "launch", "reset", "unattributed", "readinessSeconds", "controller", "simulator"], "capture");
|
||||
exactKeys(capture, ["maxMinutes", "maxActions", "maxSessions", "baselineWaitMinutes", "repack", "appearance", "launch", "reset", "unattributed", "readinessSeconds", "controller", "simulator"], "capture");
|
||||
const controller = literal(capture.controller, ["agent-device", "argent"] as const, "capture.controller", "agent-device");
|
||||
const simulator = literal(capture.simulator, ["eas", "local"] as const, "capture.simulator", "eas");
|
||||
const mode = literal(root.mode, ["deterministic", "agent"] as const, "mode", "deterministic");
|
||||
@@ -348,6 +358,7 @@ export function validateExploreConfig(value: unknown): ExploreConfig {
|
||||
maxMinutes: positiveInteger(capture.maxMinutes, "capture.maxMinutes", 30),
|
||||
maxActions: positiveInteger(capture.maxActions, "capture.maxActions", 500),
|
||||
...(capture.maxSessions === undefined ? {} : { maxSessions: positiveInteger(capture.maxSessions, "capture.maxSessions", 2) }),
|
||||
...(capture.baselineWaitMinutes === undefined ? {} : { baselineWaitMinutes: boundedInteger(capture.baselineWaitMinutes, "capture.baselineWaitMinutes", 0, 60) }),
|
||||
...(capture.repack === undefined ? {} : { repack: literal(capture.repack, ["eas", "local"] as const, "capture.repack") }),
|
||||
appearance: parseAppearance(capture.appearance, "capture.appearance"),
|
||||
launch: parseLaunchSetting(capture.launch, "capture.launch"),
|
||||
|
||||
@@ -178,7 +178,7 @@ export type SystemRegion = { kind: "status-bar" | "home-indicator"; x: number; y
|
||||
* EAS Simulator captures of both panels; the pill is the same 40×64 points on each, only its top differs. The app's own
|
||||
* rail controls start about 3% of the height below it on the outer panel, so the padding stays small.
|
||||
*/
|
||||
const DUO_PANELS = {
|
||||
export const DUO_PANELS = {
|
||||
outer: { width: 466, height: 678, statusBar: { x: 398, y: 87, width: 40, height: 64 } },
|
||||
inner: { width: 951, height: 669, statusBar: { x: 883, y: 34, width: 40, height: 64 } },
|
||||
} as const;
|
||||
@@ -251,6 +251,11 @@ export type PublicSurfaceReview = {
|
||||
description: string;
|
||||
/** Every deterministic accessibility fact for this surface, in reviewer priority order. */
|
||||
facts?: Array<{ category: string; templateId: string; bullet: string }>;
|
||||
/**
|
||||
* The screenshots match because the change is past the edge of the screen: every element its facts name reaches
|
||||
* beyond the captured screen. Only on a `modified` verdict whose screenshots match.
|
||||
*/
|
||||
offScreen?: true;
|
||||
};
|
||||
/** An evidence-bound explanation. It never changes a deterministic comparison verdict. */
|
||||
export type PublicChangeNote = {
|
||||
|
||||
@@ -57,6 +57,7 @@ function assertReviewFields(surface: Surface, enforceComparisonEvidence: boolean
|
||||
|| (["modified", "unchanged", "variant"].includes(review.status) && (!surface.base || !surface.head))
|
||||
|| (surface.sourceScope !== undefined && !["added", "removed"].includes(review.status)
|
||||
&& !SCOPE_VERDICTS[surface.sourceScope]?.includes(review.status))
|
||||
|| (review.offScreen !== undefined && (review.offScreen !== true || review.status !== "modified" || surface.visual?.verdict !== "unchanged"))
|
||||
|| (review.facts !== undefined && (!Array.isArray(review.facts) || review.facts.length > 100
|
||||
|| review.facts.some((fact) => !fact || typeof fact.category !== "string" || typeof fact.templateId !== "string"
|
||||
|| typeof fact.bullet !== "string" || !fact.bullet.trim() || fact.bullet.length > MAX_REVIEW_DESCRIPTION)))) {
|
||||
|
||||
+6
-2
@@ -6,6 +6,7 @@ import {
|
||||
type Observation,
|
||||
type Screen,
|
||||
} from "../types.js";
|
||||
import { accessibilityFingerprintsComparable } from "./store.js";
|
||||
|
||||
/**
|
||||
* Cross-run screen matching for PR-style comparisons (baseline = base branch map,
|
||||
@@ -253,7 +254,8 @@ function matchEntry(match: Match): { entry: BuildComparisonEntry; review: Compar
|
||||
// captured on both sides) hide a real change to the primary content.
|
||||
const baselineHash = match.baseline.representative.fingerprints.accessibility;
|
||||
const currentHash = match.current.representative.fingerprints.accessibility;
|
||||
const verdict = baselineHash && currentHash
|
||||
// Hashes from different fingerprint versions (an older Explore's map) say nothing about whether the trees differ.
|
||||
const verdict = baselineHash && currentHash && accessibilityFingerprintsComparable(baselineHash, currentHash)
|
||||
? (baselineHash === currentHash ? "unchanged" : "modified")
|
||||
: "unknown";
|
||||
const unchanged = verdict === "unchanged";
|
||||
@@ -262,7 +264,9 @@ function matchEntry(match: Match): { entry: BuildComparisonEntry; review: Compar
|
||||
? `Matched by ${signals}; the representative captures share an accessibility fingerprint, so the rendered tree is unchanged.`
|
||||
: verdict === "modified"
|
||||
? `Matched by ${signals}; the representative captures have different accessibility fingerprints, so the rendered tree changed.`
|
||||
: `Matched by ${signals}; accessibility fingerprints are unavailable on at least one side, so the change state could not be verified from fingerprints.`;
|
||||
: baselineHash && currentHash
|
||||
? `Matched by ${signals}; the representative captures were fingerprinted by different Explore versions, so the change state could not be verified from fingerprints.`
|
||||
: `Matched by ${signals}; accessibility fingerprints are unavailable on at least one side, so the change state could not be verified from fingerprints.`;
|
||||
const entry: BuildComparisonEntry = {
|
||||
id: `match-${match.current.screen.id}`,
|
||||
relationship: "one-to-one",
|
||||
|
||||
+23
-5
@@ -14,6 +14,7 @@ import { dirname, extname, isAbsolute, join, relative, resolve, sep } from "node
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { Ajv2020, type ErrorObject, type ValidateFunction } from "ajv/dist/2020.js";
|
||||
import { isSystemUiNode } from "../capture/controller-output.js";
|
||||
import { keyboardNodes } from "../capture/keyboard.js";
|
||||
import { validateNavigationGraph } from "../contracts/navigation-graph.js";
|
||||
import {
|
||||
CONTRACT,
|
||||
@@ -67,11 +68,26 @@ export const normalizeSelector = (input?: string): string | undefined => {
|
||||
return input.replace(/\s+/g, " ").trim() || undefined;
|
||||
};
|
||||
|
||||
const VOLATILE_SNAPSHOT_NODE_KEYS = new Set(["ref"]);
|
||||
/** A ref names a node within one read; which field has the focus can change between two reads of one screen. */
|
||||
const VOLATILE_SNAPSHOT_NODE_KEYS = new Set(["ref", "focused"]);
|
||||
|
||||
/**
|
||||
* The accessibility fingerprint's algorithm, written as its prefix. Version 2 leaves the system keyboard and the focus out
|
||||
* of the tree; a hash from before carries no prefix. Two fingerprints of different versions say nothing about whether the
|
||||
* trees differ, so a comparison treats them as not comparable rather than changed (see accessibilityFingerprintsComparable).
|
||||
*/
|
||||
const ACCESSIBILITY_FINGERPRINT_VERSION = "a2";
|
||||
|
||||
export function accessibilityFingerprintsComparable(left: string, right: string): boolean {
|
||||
const version = (hash: string) => /^(a\d+):/.exec(hash)?.[1] ?? "a1";
|
||||
return version(left) === version(right);
|
||||
}
|
||||
|
||||
/** Canonical JSON of the node tree only: controller wrappers carry per-session diagnostics and
|
||||
* ref generations that would otherwise make identical screens fingerprint differently, and Android
|
||||
* system UI nodes (status bar clock and signal, keyboard) change between emulator sessions. */
|
||||
* system UI nodes (status bar clock and signal, keyboard) change between emulator sessions. The
|
||||
* system keyboard on either platform is left out as the elements leave it out, so a keyboard up on
|
||||
* one side only does not make two captures of one screen differ. */
|
||||
function normalizeJsonSnapshotNodes(input: string): string | undefined {
|
||||
if (!/^\s*\{/.test(input)) return undefined;
|
||||
let value: unknown;
|
||||
@@ -86,10 +102,12 @@ function normalizeJsonSnapshotNodes(input: string): string | undefined {
|
||||
if (!data || typeof data !== "object" || !Array.isArray((data as { nodes?: unknown }).nodes)) {
|
||||
return undefined;
|
||||
}
|
||||
const nodes = (data as { nodes: unknown[] }).nodes;
|
||||
const keyboard: Set<unknown> = keyboardNodes(nodes.filter((entry): entry is Record<string, unknown> => !!entry && typeof entry === "object" && !Array.isArray(entry)));
|
||||
const canonical = (node: unknown): unknown => {
|
||||
if (Array.isArray(node)) {
|
||||
return node
|
||||
.filter((entry) => !(entry && typeof entry === "object" && isSystemUiNode(entry as Record<string, unknown>)))
|
||||
.filter((entry) => !(entry && typeof entry === "object" && (isSystemUiNode(entry as Record<string, unknown>) || keyboard.has(entry))))
|
||||
.map(canonical);
|
||||
}
|
||||
if (!node || typeof node !== "object") return node;
|
||||
@@ -100,7 +118,7 @@ function normalizeJsonSnapshotNodes(input: string): string | undefined {
|
||||
.map(([key, entry]) => [key, canonical(entry)]),
|
||||
);
|
||||
};
|
||||
return `nodes\0${JSON.stringify(canonical((data as { nodes: unknown[] }).nodes))}`;
|
||||
return `nodes\0${JSON.stringify(canonical(nodes))}`;
|
||||
}
|
||||
|
||||
export const normalizeSnapshot = (input: string): string =>
|
||||
@@ -118,7 +136,7 @@ export async function fingerprintState(options: {
|
||||
snapshotText?: string;
|
||||
}): Promise<ObservationFingerprints> {
|
||||
const accessibility = options.snapshotText
|
||||
? sha256(`snapshot\0${normalizeSnapshot(options.snapshotText)}`)
|
||||
? `${ACCESSIBILITY_FINGERPRINT_VERSION}:${sha256(`snapshot\0${normalizeSnapshot(options.snapshotText)}`)}`
|
||||
: undefined;
|
||||
const screenshot = options.screenshotPath ? await readFile(options.screenshotPath) : undefined;
|
||||
const visual = screenshot ? sha256(Buffer.concat([Buffer.from("visual\0"), screenshot])) : undefined;
|
||||
|
||||
+23
-8
@@ -254,24 +254,39 @@ function explorationScreenItem(context: AgentContext, map: Json, screen: Json):
|
||||
viewerUrl: ID.test(id) ? viewerUrl(context, { screen: id }) : undefined,
|
||||
});
|
||||
}
|
||||
function screenshotUrl(context: AgentContext, map: Json, screen: Json): string | undefined {
|
||||
/** The screenshot a screen is shown by: its latest observation's. */
|
||||
function screenshotArtifact(map: Json, screen: Json): Json | undefined {
|
||||
const observationId = strings(screen.observationIds).at(-1);
|
||||
const observation = arr(map.observations).map(rec).find(o => o.id === observationId);
|
||||
const artifact = arr(map.artifacts).map(rec).find(a => a.id === observation?.screenshotArtifactId);
|
||||
const sha = str(artifact?.sha256);
|
||||
return arr(map.artifacts).map(rec).find(a => a.id === observation?.screenshotArtifactId);
|
||||
}
|
||||
function screenshotUrl(context: AgentContext, map: Json, screen: Json): string | undefined {
|
||||
const sha = str(screenshotArtifact(map, screen)?.sha256);
|
||||
return sha && SHA256.test(sha) ? withPlatform(context, apiUrl(context, `/assets/${sha}`)) : undefined;
|
||||
}
|
||||
const elementItem = (element: Json): Json => defined({
|
||||
id: str(element.id ?? element.elementId), label: str(element.label), role: str(element.role), value: str(element.value), identifier: str(element.identifier),
|
||||
bounds: element.bounds === undefined ? undefined : rec(element.bounds), actions: strings(element.actions), risk: str(element.risk), actionable: bool(element.actionable), enabled: bool(element.enabled), visible: bool(element.visible),
|
||||
});
|
||||
/**
|
||||
* One element, its bounds on the axes of the screenshot it is listed beside. A screenshot stored turned a quarter
|
||||
* clockwise (an opened iPhone Duo's) is `turnedHeight` points tall on the tree's axes, where the map keeps the bounds:
|
||||
* there the portrait point (x, y) shows at (H - y, x).
|
||||
*/
|
||||
const elementItem = (element: Json, turnedHeight?: number): Json => {
|
||||
const bounds = element.bounds === undefined ? undefined : rec(element.bounds);
|
||||
const [x, y, width, height] = [num(bounds?.x), num(bounds?.y), num(bounds?.width), num(bounds?.height)];
|
||||
const shown = bounds && turnedHeight !== undefined && bounds.unit === "points" && x !== undefined && y !== undefined && width !== undefined && height !== undefined
|
||||
? { x: turnedHeight - y - height, y: x, width: height, height: width, unit: "points" } : bounds;
|
||||
return defined({
|
||||
id: str(element.id ?? element.elementId), label: str(element.label), role: str(element.role), value: str(element.value), identifier: str(element.identifier),
|
||||
bounds: shown, actions: strings(element.actions), risk: str(element.risk), actionable: bool(element.actionable), enabled: bool(element.enabled), visible: bool(element.visible),
|
||||
});
|
||||
};
|
||||
const transitionItem = (transition: Json): Json => defined({
|
||||
id: str(transition.id), from: str(transition.fromScreenId), to: str(transition.toScreenId), actionKind: str(transition.actionKind), elementId: str(transition.elementId), attempts: strings(transition.attemptIds).length,
|
||||
});
|
||||
function explorationScreenDetail(context: AgentContext, map: Json, screen: Json): Json {
|
||||
const id = str(screen.id);
|
||||
const transitions = arr(map.transitions).map(rec);
|
||||
const elements = arr(map.elements).map(rec).filter(element => element.screenId === id).map(elementItem);
|
||||
const turnedHeight = num(rec(screenshotArtifact(map, screen)?.turned).height);
|
||||
const elements = arr(map.elements).map(rec).filter(element => element.screenId === id).map(element => elementItem(element, turnedHeight));
|
||||
const variants = arr(map.variants).map(rec).filter(variant => variant.screenId === id).map(variant => defined({ id: str(variant.id), key: str(variant.key), name: str(variant.name), observations: strings(variant.observationIds).length }));
|
||||
const gaps = arr(map.gaps).map(rec).filter(gap => gap.screenId === id).map(gapItem);
|
||||
return defined({
|
||||
|
||||
@@ -143,6 +143,16 @@ function notComparedChange(surface: Surface): boolean {
|
||||
return verified && (!surface.review || surface.visual?.reason === "diff-failed");
|
||||
}
|
||||
|
||||
/**
|
||||
* An accessibility-only change whose elements lie past the edge of the screen (the review's `offScreen`), so another
|
||||
* target's row can say the change is there, off-screen, rather than missing. Like notComparedChange, a comparison that
|
||||
* failed or a redirect is not one.
|
||||
*/
|
||||
function offScreenChange(surface: Surface): boolean {
|
||||
return surface.review?.offScreen === true && isAccessibilityOnlyChange(surface) && surface.visual?.reason !== "diff-failed"
|
||||
&& (surface.head ?? surface.base)?.kind !== "redirect";
|
||||
}
|
||||
|
||||
function routeOf(surface: Surface): string {
|
||||
return plainText(surface.head?.route || surface.base?.route || surface.id, MAX_ROUTE_LENGTH);
|
||||
}
|
||||
@@ -302,7 +312,8 @@ export function platformRecordOf(state: PublicCommentState, target?: CommentTarg
|
||||
const group = groupOf(surface, evidenceBased);
|
||||
if (group === "unlisted") continue;
|
||||
const scheme = schemeOf(surface.id);
|
||||
record.changes.push({ id: surface.id, route: routeOf(surface), group, description: descriptionOf(surface, group), ...(scheme ? { scheme } : {}) });
|
||||
record.changes.push({ id: surface.id, route: routeOf(surface), group, description: descriptionOf(surface, group), ...(scheme ? { scheme } : {}),
|
||||
...(group === "accessibility-only" && offScreenChange(surface) ? { offScreen: true as const } : {}) });
|
||||
}
|
||||
const navigation = state.map ? navigationOf(state.review, new Map(record.changes.map(change => [change.id, change.group]))) : undefined;
|
||||
if (navigation) record.navigation = navigation;
|
||||
@@ -562,10 +573,24 @@ function renderChangeRow(row: ChangeRow, ready: PlatformRecord[]): string {
|
||||
text = `${route} · ${description}`;
|
||||
}
|
||||
const tag = row.group === "accessibility-only" ? "accessibility only" : row.group === "needs-review" ? "needs review" : undefined;
|
||||
return `- ${[text, tag, ...whereNote(row, ready)].filter(Boolean).join(" · ")}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Where a screen's change shows. A target whose screenshots match because the changed elements lie past the edge of the
|
||||
* screen has the change, off-screen, so it is named apart: "iPhone 17, Android only · off-screen on iPhone Duo (closed)"
|
||||
* for Tempo's footnote below the fold of its sheet in brentvatne/euxy#172 (review run 39362642343). The targets that show
|
||||
* it are still named the shorter way among the others (none when all of them do); without such a target the note is what
|
||||
* it always was.
|
||||
*/
|
||||
function whereNote(row: ChangeRow, ready: PlatformRecord[]): string[] {
|
||||
const offScreen = row.group === "modified"
|
||||
? row.targets.filter(record => record.changes?.some(change => change.id === row.id && change.offScreen === true)) : [];
|
||||
const shown = row.targets.filter(record => !offScreen.includes(record));
|
||||
// A target with screens it has no verdict on may have changed this one too: name where it changed, not where it did not.
|
||||
const where = ready.some(record => !row.targets.includes(record) && gaps(record).length)
|
||||
? `on ${row.targets.map(labelOf).join(", ")}` : targetsNote(row.targets, ready);
|
||||
return `- ${[text, tag, where].filter(Boolean).join(" · ")}`;
|
||||
const unsure = ready.some(record => !row.targets.includes(record) && gaps(record).length);
|
||||
const visible = unsure ? `on ${shown.map(labelOf).join(", ")}` : targetsNote(shown, ready.filter(record => !offScreen.includes(record)));
|
||||
return [visible, offScreen.length ? `off-screen on ${offScreen.map(labelOf).join(", ")}` : undefined].filter((part): part is string => Boolean(part));
|
||||
}
|
||||
|
||||
/** "Android only" / "not on Android": the targets a screen changed on, when not all of them, named the shorter way. */
|
||||
|
||||
@@ -103,8 +103,11 @@ export type PlatformRecord = {
|
||||
accessibility?: { compared: number; described: number; failed: number };
|
||||
routes?: number;
|
||||
};
|
||||
/** The review's changed surfaces, one per surface; `scheme` for a secondary appearance (`<route>@<scheme>`). */
|
||||
changes?: { id: string; route: string; group: ChangeGroup; description: string; scheme?: string }[];
|
||||
/**
|
||||
* The review's changed surfaces, one per surface; `scheme` for a secondary appearance (`<route>@<scheme>`). `offScreen`
|
||||
* marks an accessibility-only change whose elements lie past the edge of the screen.
|
||||
*/
|
||||
changes?: { id: string; route: string; group: ChangeGroup; description: string; scheme?: string; offScreen?: true }[];
|
||||
/** Screens whose source changed and whose captures were verified, but that got no verdict: the comparison did not run or failed. */
|
||||
notCompared?: number;
|
||||
/** Differences stated once rather than per screen (variants, app-wide source changes). */
|
||||
@@ -270,7 +273,8 @@ export function parsePlatformRecord(value: unknown): PlatformRecord | undefined
|
||||
if (Array.isArray(value.changes)) {
|
||||
const changes = value.changes.flatMap((entry): NonNullable<PlatformRecord["changes"]> => isRecord(entry) && isString(entry.id, 220) && isString(entry.route, 2000)
|
||||
&& CHANGE_GROUPS.includes(entry.group as ChangeGroup) && isString(entry.description, 2000)
|
||||
? [{ id: entry.id, route: entry.route, group: entry.group as ChangeGroup, description: entry.description, ...(isString(entry.scheme, 20) ? { scheme: entry.scheme } : {}) }] : []);
|
||||
? [{ id: entry.id, route: entry.route, group: entry.group as ChangeGroup, description: entry.description, ...(isString(entry.scheme, 20) ? { scheme: entry.scheme } : {}),
|
||||
...(entry.offScreen === true && entry.group === "accessibility-only" ? { offScreen: true as const } : {}) }] : []);
|
||||
if (changes.length <= 2000) record.changes = changes;
|
||||
}
|
||||
const navigation = parseNavigation(value.navigation);
|
||||
|
||||
@@ -716,6 +716,7 @@ export async function assessPublicEvidence(options: {
|
||||
const summary = summarizeReviewerChanges({ changes: result.reviewerChanges });
|
||||
accessibilityCounts.compared += 1;
|
||||
review = decideReview(scope, visual, summary);
|
||||
if (result.offScreen && review.status === "modified" && visual.verdict === "unchanged") review = { ...review, offScreen: true };
|
||||
if (review.facts?.length) accessibilityCounts.described += 1;
|
||||
} catch (error) {
|
||||
accessibilityCounts.failed += 1;
|
||||
|
||||
@@ -93,6 +93,8 @@ export type EvidenceComparatorResult = {
|
||||
visualAssessment?: VisualEvidenceAssessment;
|
||||
/** Feed directly to summarizeReviewerChanges. */
|
||||
reviewerChanges: ReviewerChangeInput[];
|
||||
/** The screenshots match, and every element the changes name lies at least partly past the edge of the screen. */
|
||||
offScreen?: true;
|
||||
};
|
||||
|
||||
type SupportedInput = Extract<ReviewerChangeInput, { assessment: "supported" }>;
|
||||
@@ -617,6 +619,54 @@ function outermost(elements: readonly ObservedElement[]): ObservedElement[] {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The screen in its accessibility tree's units: the element at the origin with the screenshot's shape (either way round,
|
||||
* since a frame turned to show an opened iPhone Duo upright keeps the tree on the panel's portrait axes), the largest if
|
||||
* several. Both controllers report the app's root that way. Without the screenshot's size there is no telling which
|
||||
* element is the screen.
|
||||
*/
|
||||
function viewportOf(side: ResolvedSide): ElementBounds | undefined {
|
||||
const image = side.map.artifacts.find(artifact => artifact.id === side.observation.screenshotArtifactId);
|
||||
if (!image?.width || !image.height) return undefined;
|
||||
const { width, height } = image;
|
||||
const shaped = (bounds: ElementBounds) => Math.abs(bounds.width * height - bounds.height * width) <= 0.01 * width * height
|
||||
|| Math.abs(bounds.width * width - bounds.height * height) <= 0.01 * width * height;
|
||||
return side.observation.elements.flatMap(element => boundsOf(element) ?? [])
|
||||
.filter(bounds => Math.abs(bounds.x) < 1 && Math.abs(bounds.y) < 1 && bounds.width > 0 && bounds.height > 0 && shaped(bounds))
|
||||
.sort((a, b) => b.width * b.height - a.width * a.height)[0];
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the screenshots cannot show the changes because the elements they name lie past the edge of the screen: each
|
||||
* change names at least one element, and more than half of each named element's area is beyond the screen on its side.
|
||||
* Both controllers leave an element wholly outside the screen out of the tree (the Argent bridge as agent-device does),
|
||||
* so a change past the edge can only show as an element straddling it, as Tempo's footnote below the fold of its sheet
|
||||
* on the closed iPhone Duo in brentvatne/euxy#172 (review run 39362642343) would. Mostly outside, not merely crossing
|
||||
* an edge: a control nudged a point or two past the edge is still on the screen, and its change stays "accessibility
|
||||
* only". A change whose elements have no bounds never qualifies.
|
||||
*/
|
||||
function changesOffScreen(facts: readonly SupportedInput[], base: ResolvedSide, head: ResolvedSide): boolean {
|
||||
const changes = facts.filter(fact => !["unstable-text", "no-visible-change", "visual-changed"].includes(fact.change.kind));
|
||||
if (!changes.length) return false;
|
||||
const viewports = { base: viewportOf(base), head: viewportOf(head) };
|
||||
const outside = (reference: string) => {
|
||||
const [, side, id] = /^(base|head):element:(.+)$/.exec(reference) ?? [];
|
||||
if (side !== "base" && side !== "head") return undefined;
|
||||
const viewport = viewports[side];
|
||||
const element = (side === "base" ? base : head).observation.elements.find(candidate => candidate.elementId === id);
|
||||
const bounds = element && boundsOf(element);
|
||||
if (!viewport || !bounds) return false;
|
||||
const visibleWidth = Math.max(0, Math.min(bounds.x + bounds.width, viewport.x + viewport.width) - Math.max(bounds.x, viewport.x));
|
||||
const visibleHeight = Math.max(0, Math.min(bounds.y + bounds.height, viewport.y + viewport.height) - Math.max(bounds.y, viewport.y));
|
||||
const area = bounds.width * bounds.height;
|
||||
return area > 0 && visibleWidth * visibleHeight < area / 2;
|
||||
};
|
||||
return changes.every(fact => {
|
||||
const named = fact.evidenceRefs.map(outside).filter((value): value is boolean => value !== undefined);
|
||||
return named.length > 0 && named.every(Boolean);
|
||||
});
|
||||
}
|
||||
|
||||
function specificByLabel(elements: readonly ObservedElement[]): ObservedElement[] {
|
||||
const byLabel = new Map<string, ObservedElement>();
|
||||
const unlabelled: ObservedElement[] = [];
|
||||
@@ -1276,6 +1326,7 @@ function commonResult(input: CommonSurfaceComparatorInput): EvidenceComparatorRe
|
||||
const status: "modified" | "unchanged" = hasRuntimeDelta || reasons.length > 0
|
||||
? "modified"
|
||||
: "unchanged";
|
||||
const offScreen = input.visualAssessment.status === "unchanged" && changesOffScreen(facts, base, head);
|
||||
return {
|
||||
surfaceId: input.impact.id,
|
||||
status,
|
||||
@@ -1288,6 +1339,7 @@ function commonResult(input: CommonSurfaceComparatorInput): EvidenceComparatorRe
|
||||
),
|
||||
visualAssessment: input.visualAssessment,
|
||||
reviewerChanges,
|
||||
...(offScreen ? { offScreen: true as const } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -144,3 +144,39 @@ export function assertDecodedPng(contents: Buffer, label: string): void {
|
||||
if (decoded[row * (rowBytes + 1)]! > 4) throw new Error(`${label} has an invalid PNG row filter`);
|
||||
}
|
||||
}
|
||||
|
||||
/** The chunks that say how a PNG's samples map to colour; pngjs reads past them and writes none of its own. */
|
||||
const COLOUR_CHUNKS = new Set(["iCCP", "sRGB", "gAMA", "cHRM"]);
|
||||
|
||||
/**
|
||||
* The image turned a quarter clockwise. An opened iPhone Duo shows a landscape UI on a panel whose native axes are
|
||||
* portrait, and its screenshot comes back on those axes; turned, it is what a person sees. The source's colour chunks
|
||||
* are carried over unchanged. pngjs reads every PNG as 8-bit RGBA, so a 16-bit image comes out 8-bit; simulator
|
||||
* screenshots are 8-bit already.
|
||||
*/
|
||||
export function turnedClockwise(png: Buffer): Buffer {
|
||||
const { PNG } = createRequire(import.meta.url.replace(/\?.*$/, ""))("pngjs") as { PNG: { new(size: { width: number; height: number }): DecodedFrame; sync: {
|
||||
read(bytes: Buffer, options: { checkCRC: boolean }): DecodedFrame; write(frame: DecodedFrame): Buffer } } };
|
||||
const source = PNG.sync.read(png, { checkCRC: false });
|
||||
const { width, height } = source;
|
||||
const turned = new PNG({ width: height, height: width });
|
||||
for (let y = 0; y < height; y += 1) {
|
||||
for (let x = 0; x < width; x += 1) {
|
||||
const from = (y * width + x) * 4;
|
||||
source.data.copy(turned.data, (x * height + (height - 1 - y)) * 4, from, from + 4);
|
||||
}
|
||||
}
|
||||
const written = PNG.sync.write(turned);
|
||||
// Each chunk is copied whole, CRC included: its data does not depend on the pixels or their size. They belong before
|
||||
// the image data, so they go right after the header (8 bytes of signature, then IHDR's 25). The output is RGBA, so a
|
||||
// grey source's ICC profile, which describes grey samples only, is left out.
|
||||
const grey = png[25] === 0 || png[25] === 4;
|
||||
const colour: Buffer[] = [];
|
||||
for (let offset = 8; offset + 12 <= png.length;) {
|
||||
const length = png.readUInt32BE(offset), type = png.toString("latin1", offset + 4, offset + 8), next = offset + 12 + length;
|
||||
if (type === "IDAT" || type === "IEND" || next > png.length) break;
|
||||
if (COLOUR_CHUNKS.has(type) && !(grey && type === "iCCP")) colour.push(png.subarray(offset, next));
|
||||
offset = next;
|
||||
}
|
||||
return colour.length ? Buffer.concat([written.subarray(0, 33), ...colour, written.subarray(33)]) : written;
|
||||
}
|
||||
@@ -121,6 +121,12 @@ export type Artifact = {
|
||||
mimeType: string;
|
||||
width?: number;
|
||||
height?: number;
|
||||
/**
|
||||
* A screenshot stored turned a quarter clockwise from its accessibility tree's axes (an opened iPhone Duo's landscape UI
|
||||
* on its inner panel's portrait axes): the tree's screen, `width` × `height` points. Element bounds stay on the tree's
|
||||
* axes, where the portrait point (x, y) shows at (height - y, x).
|
||||
*/
|
||||
turned?: { width: number; height: number };
|
||||
createdAt: string;
|
||||
};
|
||||
|
||||
|
||||
@@ -143,6 +143,10 @@ jobs:
|
||||
set -euo pipefail
|
||||
npm ci --ignore-scripts --prefix "app-base/$APP_INSTALL_DIR"
|
||||
npm ci --ignore-scripts --prefix "app-head/$APP_INSTALL_DIR"
|
||||
# A finished baseline of the base commit is downloaded here. With none, ci:find-baseline names a baseline run still
|
||||
# capturing it (pending): right after a merge the next review starts while main's baseline runs. Capture review
|
||||
# checks that run again (this repository's baseline workflow, the default branch, a push or manual dispatch, exactly
|
||||
# the base commit) and waits for its artifact while head captures, for up to capture.baselineWaitMinutes.
|
||||
- name: Find optional exact-base evidence
|
||||
id: cache
|
||||
continue-on-error: true
|
||||
@@ -164,6 +168,8 @@ jobs:
|
||||
id: capture
|
||||
env:
|
||||
COMPARE: ${{ needs.targets.outputs.compare }}
|
||||
# Reads the pending baseline run's jobs and artifact (actions: read); a comparison makes no other GitHub request.
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
SCOPE: ${{ inputs.scope }}
|
||||
EXPIRES_IN: ${{ inputs.expires-in }}
|
||||
EXPO_TOKEN: ${{ secrets.EXPO_TOKEN }}
|
||||
@@ -172,6 +178,7 @@ jobs:
|
||||
ANTHROPIC_WORKSPACE_ID: ${{ vars.ANTHROPIC_WORKSPACE_ID }}
|
||||
# App-test secrets: one NAME: line per {{secret:NAME}} input in .expo-explore (expo-explore setup adds them).
|
||||
EXPLORE_CAPTURE_CACHE: ${{ runner.temp }}/explore-cache
|
||||
EXPLORE_BASELINE_RUN: ${{ steps.cache.outputs.pending }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
@@ -133,6 +133,10 @@ jobs:
|
||||
set -euo pipefail
|
||||
npm ci --ignore-scripts --prefix "app-base/$APP_INSTALL_DIR"
|
||||
npm ci --ignore-scripts --prefix "app-head/$APP_INSTALL_DIR"
|
||||
# A finished baseline of the base commit is downloaded here. With none, ci:find-baseline names a baseline run still
|
||||
# capturing it (pending): right after a merge the next review starts while main's baseline runs. Capture review
|
||||
# checks that run again (this repository's baseline workflow, the default branch, a push or manual dispatch, exactly
|
||||
# the base commit) and waits for its artifact while head captures, for up to capture.baselineWaitMinutes.
|
||||
- name: Find optional exact-base evidence
|
||||
id: cache
|
||||
continue-on-error: true
|
||||
@@ -161,6 +165,7 @@ jobs:
|
||||
ANTHROPIC_WORKSPACE_ID: ${{ vars.ANTHROPIC_WORKSPACE_ID }}
|
||||
# App-test secrets: one NAME: line per {{secret:NAME}} input in .expo-explore (expo-explore setup adds them).
|
||||
EXPLORE_CAPTURE_CACHE: ${{ runner.temp }}/explore-cache
|
||||
EXPLORE_BASELINE_RUN: ${{ steps.cache.outputs.pending }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
@@ -169,6 +169,8 @@ const Navigation=createStaticNavigation(Root); export default function App(){ret
|
||||
}
|
||||
if (command === 'simulator:get') { const session = sessions.get(value(request, '--id')); return json({ id: session.id, name: session.name, type: 'agent-device', platform: platform.toUpperCase(), status: 'IN_PROGRESS', app: { id: PROJECT_ID, slug: 'example', ownerAccount: { name: 'acme' } }, remoteConfig: { agentDeviceRemoteSessionUrl: `https://controller.example/${session.id}`, agentDeviceRemoteSessionToken: 'controller-private-token' } }); }
|
||||
if (command === 'simulator:stop') return json({ id: value(request, '--id'), status: 'STOPPED' });
|
||||
// The session's serve-sim offers no simulator:control actions, so the capture falls back to the controller's own.
|
||||
if (command === 'simulator:control' && request.args.includes('--list')) return json({});
|
||||
}
|
||||
if (basename(request.command) === 'repack-app') {
|
||||
repackedSha = git(request.cwd, 'rev-parse', 'HEAD');
|
||||
|
||||
@@ -403,6 +403,24 @@ test("review captures and reports each configured target in its own matrix job,
|
||||
}
|
||||
});
|
||||
|
||||
test("a review waits for the baseline run ci:find-baseline names, with the token's Actions read and nothing the PR head chooses", () => {
|
||||
for (const workflow of [review, compare]) {
|
||||
const steps = workflow.jobs.capture.steps;
|
||||
const find = steps.find(step => step.name === "Find optional exact-base evidence");
|
||||
const capture = steps.find(step => step.name === "Capture review");
|
||||
assert.equal(find["continue-on-error"], true, "a failed lookup never fails the review");
|
||||
assert.ok(steps.indexOf(find) < steps.indexOf(capture));
|
||||
// The run is the find step's output, chosen by the base commit and the default branch's baseline workflow; capture
|
||||
// checks it again before it waits.
|
||||
assert.equal(capture.env.EXPLORE_BASELINE_RUN, "${{ steps.cache.outputs.pending }}");
|
||||
assert.equal(capture.env.GITHUB_TOKEN, "${{ github.token }}", "the wait reads the run, its jobs and its artifact");
|
||||
assert.equal(workflow.jobs.capture.permissions.actions, "read");
|
||||
assert.equal(workflow.jobs.capture.permissions.contents, "read");
|
||||
}
|
||||
assert.deepEqual(compare.jobs.capture.permissions, { contents: "read", actions: "read" }, "the wait needs no permission a comparison did not have");
|
||||
assert.deepEqual(review.jobs.capture.permissions, { contents: "read", actions: "read", "pull-requests": "write" });
|
||||
});
|
||||
|
||||
test("compare reviews any two refs from a manual run or another workflow, with no pull request access", () => {
|
||||
for (const trigger of ["workflow_dispatch", "workflow_call"]) {
|
||||
assert.deepEqual(Object.keys(compare.on[trigger].inputs).sort(), ["base", "expires-in", "head", "scope"]);
|
||||
|
||||
@@ -201,6 +201,8 @@ test("Android fill uses Argent's fill and requires it to verify the text", async
|
||||
const snapshot = await run(["snapshot", ...flags("android")]);
|
||||
const node = snapshot.json.data.nodes.find(item => item.type === "TextField");
|
||||
assert.ok(node, "the rename fixture has an EditText");
|
||||
assert.equal(node.focused, true, "the focused field is marked, as agent-device marks it");
|
||||
assert.equal(snapshot.json.data.keyboardVisible, false, "whether a keyboard shows, which the Android tree cannot say, is passed on");
|
||||
assert.equal((await run(["fill", node.ref, ...flags("android"), "--", "Groove"])).exitCode, 0);
|
||||
assert.deepEqual(calls.find(call => call.tool === "fill").args, { udid: "emulator-5554", text: "Groove" });
|
||||
const unverified = await run(["fill", node.ref, ...flags("android"), "--", "Groove"]);
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
import { readFile, readdir } from 'node:fs/promises';
|
||||
import { join } from 'node:path';
|
||||
import { Uint8ArrayReader, Uint8ArrayWriter, ZipWriter, configure } from '@zip.js/zip.js';
|
||||
import { runPublicCi, runBaselineCi, captureBaseAfterWait } from '../../dist/ci/pipeline.js';
|
||||
import { readCaptureBundle } from '../../dist/review/capture-review.js';
|
||||
import { fixture } from '../helpers/public-ci-fixture.mjs';
|
||||
|
||||
const deferred = () => { let resolve, reject; const promise = new Promise((done, fail) => { resolve = done; reject = fail; }); return { promise, resolve, reject }; };
|
||||
const tick = () => new Promise(done => setImmediate(done));
|
||||
|
||||
/** The files of a baseline export, zipped as actions/upload-artifact uploads them. */
|
||||
async function zipDirectory(directory) {
|
||||
configure({ useWebWorkers: false });
|
||||
const writer = new ZipWriter(new Uint8ArrayWriter(), { useWebWorkers: false });
|
||||
for (const entry of await readdir(directory, { recursive: true, withFileTypes: true })) {
|
||||
if (!entry.isFile()) continue;
|
||||
const path = join(entry.parentPath, entry.name);
|
||||
await writer.add(path.slice(directory.length + 1), new Uint8ArrayReader(await readFile(path)));
|
||||
}
|
||||
return writer.close();
|
||||
}
|
||||
|
||||
test('base starts the moment the wait gives up, beside head, and a waited baseline is restored only once head has finished', async () => {
|
||||
const events = [];
|
||||
const run = ({ head, waited }) => captureBaseAfterWait({ waited: waited?.promise, head: head.promise, stopWaiting: () => events.push('stop waiting'),
|
||||
captureBase: async cache => { events.push(cache ? `base from ${cache}` : 'base captures'); } });
|
||||
// The baseline job failed while head was still capturing: base captures at once, in its own session.
|
||||
let head = deferred(), waited = deferred();
|
||||
let base = run({ head, waited });
|
||||
waited.resolve(undefined); await tick();
|
||||
assert.deepEqual(events, ['base captures'], 'base did not wait for head');
|
||||
head.resolve(); await base;
|
||||
// The baseline arrived first: restoring it needs head's runtime profile, so base starts after head.
|
||||
events.length = 0; head = deferred(); waited = deferred();
|
||||
base = run({ head, waited });
|
||||
waited.resolve('/runner/baseline-download'); await tick();
|
||||
assert.deepEqual(events, []);
|
||||
head.resolve(); await base;
|
||||
assert.deepEqual(events, ['base from /runner/baseline-download']);
|
||||
// Head failed while base still waited: the wait stops, and base never starts for a failed review.
|
||||
events.length = 0; head = deferred(); waited = deferred();
|
||||
base = run({ head, waited });
|
||||
head.reject(new Error('head capture failed')); await tick();
|
||||
assert.deepEqual(events, ['stop waiting']);
|
||||
waited.resolve(undefined); await base;
|
||||
assert.deepEqual(events, ['stop waiting'], 'base was not captured');
|
||||
// Without a wait, base starts as before.
|
||||
events.length = 0;
|
||||
await captureBaseAfterWait({ waited: undefined, head: new Promise(() => {}), stopWaiting: () => events.push('stop waiting'), captureBase: async () => { events.push('base captures'); } });
|
||||
assert.deepEqual(events, ['base captures']);
|
||||
});
|
||||
|
||||
test('head capture starts while the review waits for the baseline, and the waited baseline is restored for base', async t => {
|
||||
const f = await fixture(t, 'expo-router');
|
||||
t.mock.method(globalThis, 'fetch', async input => assert.fail(`Unmocked network request: ${input}`));
|
||||
const baseline = await runBaselineCi({ project: f.baseProject, sha: f.baseSha, output: join(f.root, 'baseline'), config: f.config, environment: f.environment, runner: f.runner, now: f.now, sleep: f.sleep });
|
||||
const before = f.requests.length;
|
||||
// The baseline finishes only once head capture has started a simulator session; were head held back by the wait,
|
||||
// the fallback would end it with no baseline after 20 s, and base would capture.
|
||||
const finished = deferred();
|
||||
let finishedBy;
|
||||
const fallback = setTimeout(() => { finishedBy ??= 'timeout'; finished.resolve(undefined); }, 20_000);
|
||||
t.after(() => clearTimeout(fallback));
|
||||
const runner = request => {
|
||||
if (request.args[0] === 'simulator:start' && !finishedBy) { finishedBy = 'head started'; setImmediate(() => finished.resolve(baseline.directory)); }
|
||||
return f.runner(request);
|
||||
};
|
||||
let waitSignal;
|
||||
const progress = [];
|
||||
const output = join(f.root, 'review');
|
||||
await runPublicCi({ baseProject: f.baseProject, headProject: f.headProject, pullRequestPath: f.pullRequestPath, output, config: f.config,
|
||||
environment: f.environment, transport: f.transport, runner, now: f.now, sleep: f.sleep, progress: line => progress.push(line),
|
||||
awaitBaseline: signal => { waitSignal = signal; return finished.promise; } });
|
||||
assert.equal(finishedBy, 'head started');
|
||||
assert.equal(f.requests.slice(before).filter(request => request.args[0] === 'simulator:start').length, 1, 'base reused the baseline instead of a second session');
|
||||
assert.ok(progress.includes(`base waits for the baseline run of ${f.baseSha.slice(0, 8)} still being captured; head captures meanwhile`));
|
||||
assert.ok(progress.includes(`base: found the baseline captures of ${f.baseSha.slice(0, 8)}`));
|
||||
const baselineBundle = await readCaptureBundle(baseline.directory);
|
||||
for (const [id, image] of Object.entries((await readCaptureBundle(join(output, 'base-export'))).images)) assert.deepEqual(image, baselineBundle.images[id]);
|
||||
assert.equal(JSON.parse(await readFile(join(output, 'execution.json'), 'utf8')).capture.baseFromCache, true);
|
||||
assert.equal(waitSignal.aborted, true, 'the wait is told to stop once the review is done');
|
||||
});
|
||||
|
||||
test('capture:review waits for the baseline run ci:find-baseline named, downloads its artifact, and restores base from it', async t => {
|
||||
const f = await fixture(t, 'expo-router');
|
||||
t.mock.method(globalThis, 'fetch', async input => assert.fail(`Unmocked network request: ${input}`));
|
||||
const baseline = await runBaselineCi({ project: f.baseProject, sha: f.baseSha, output: join(f.root, 'baseline'), config: f.config, environment: f.environment, runner: f.runner, now: f.now, sleep: f.sleep });
|
||||
const zip = await zipDirectory(baseline.directory);
|
||||
const actions = [];
|
||||
const transport = async (input, init = {}) => {
|
||||
const url = new URL(input);
|
||||
if (url.origin === 'https://blob.example') { actions.push('download'); assert.equal(init.headers?.authorization, undefined); return new Response(zip); }
|
||||
const path = url.pathname.replace('/repos/acme/example', '');
|
||||
if (path === '' || path.startsWith('/actions/runs/40') || path.startsWith('/actions/workflows/') || path.startsWith('/actions/artifacts/77')) actions.push(path);
|
||||
if (path === '') return Response.json({ default_branch: 'main' });
|
||||
if (path === '/actions/workflows/explore-baseline.yml') return Response.json({ id: 5 });
|
||||
if (path === '/actions/runs/40') return Response.json({ id: 40, status: 'in_progress', conclusion: null, workflow_id: 5, head_sha: f.baseSha, head_branch: 'main', event: 'push', head_repository: { full_name: 'acme/example' } });
|
||||
if (path === '/actions/runs/40/artifacts') return Response.json({ artifacts: [{ id: 77, name: url.searchParams.get('name'), expired: false, workflow_run: { id: 40 } }] });
|
||||
if (path === '/actions/artifacts/77/zip') return new Response(null, { status: 302, headers: { location: 'https://blob.example/77.zip' } });
|
||||
return f.transport(input, init);
|
||||
};
|
||||
const review = async (name, config) => {
|
||||
const before = f.requests.length; actions.length = 0;
|
||||
const output = join(f.root, name);
|
||||
const progress = [];
|
||||
await runPublicCi({ baseProject: f.baseProject, headProject: f.headProject, pullRequestPath: f.pullRequestPath, output, config,
|
||||
environment: { ...f.environment, EXPLORE_BASELINE_RUN: '40' }, transport, runner: f.runner, now: f.now, sleep: f.sleep, progress: line => progress.push(line) });
|
||||
return { output, progress, sessions: f.requests.slice(before).filter(request => request.args[0] === 'simulator:start').length };
|
||||
};
|
||||
const waited = await review('review', f.config);
|
||||
assert.deepEqual(actions, ['', '/actions/workflows/explore-baseline.yml', '/actions/runs/40', '/actions/runs/40/artifacts', '/actions/artifacts/77/zip', 'download']);
|
||||
assert.ok(waited.progress.includes(`base: downloaded the baseline of ${f.baseSha.slice(0, 8)} that run 40 captured`));
|
||||
assert.equal(waited.sessions, 1, 'base was restored from the downloaded baseline');
|
||||
assert.equal(JSON.parse(await readFile(join(waited.output, 'execution.json'), 'utf8')).capture.baseFromCache, true);
|
||||
const off = await review('review-no-wait', { ...f.config, capture: { ...f.config.capture, baselineWaitMinutes: 0 } });
|
||||
assert.deepEqual(actions, [], 'capture.baselineWaitMinutes: 0 never waits');
|
||||
assert.equal(off.sessions, 2, 'base captured itself');
|
||||
});
|
||||
@@ -0,0 +1,203 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, readFile, readdir, rm, stat } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import test from "node:test";
|
||||
import { Uint8ArrayReader, Uint8ArrayWriter, ZipWriter, configure } from "@zip.js/zip.js";
|
||||
|
||||
const { waitForBaselineArtifact, awaitPendingBaseline } = await import("../../dist/ci/baseline-wait.js");
|
||||
const { findPendingBaselineRun, findOptionalBaseline } = await import("../../dist/ci/workflow-steps.js");
|
||||
|
||||
const sha = "a".repeat(40);
|
||||
const environment = { GITHUB_REPOSITORY: "acme/app", GITHUB_TOKEN: "test-token" };
|
||||
const trustedRun = { id: 20, workflow_id: 77, head_sha: sha, head_branch: "main", event: "push", head_repository: { full_name: "acme/app" } };
|
||||
const job = (status, conclusion = null, target = "iphone") => ({ name: `baseline (${target})`, status, conclusion });
|
||||
|
||||
async function zipFiles(files) {
|
||||
configure({ useWebWorkers: false });
|
||||
const writer = new ZipWriter(new Uint8ArrayWriter(), { useWebWorkers: false });
|
||||
for (const [name, content] of Object.entries(files)) await writer.add(name, new Uint8ArrayReader(new TextEncoder().encode(content)));
|
||||
return writer.close();
|
||||
}
|
||||
|
||||
/**
|
||||
* GitHub as a review's wait sees it: baseline run 20 of `sha`, whose state moves to the next of `states` each time the
|
||||
* run is read (the last one stays), and artifact 500 of the iphone target, served from a storage URL GitHub redirects to.
|
||||
*/
|
||||
function fakeGitHub({ states = [{}], run = {}, zip, runFailures = [], runs } = {}) {
|
||||
let poll = -1;
|
||||
const requests = [];
|
||||
const state = () => states[Math.min(Math.max(poll, 0), states.length - 1)];
|
||||
const transport = async (input, init = {}) => {
|
||||
const url = new URL(input);
|
||||
requests.push(`${url.pathname}${url.search}`);
|
||||
if (url.origin === "https://blob.example") {
|
||||
assert.equal(init.headers?.authorization, undefined, "the token never reaches the storage URL GitHub redirects to");
|
||||
return new Response(zip);
|
||||
}
|
||||
assert.equal(url.origin, "https://api.github.com");
|
||||
assert.equal(init.headers?.authorization, "Bearer test-token");
|
||||
const path = url.pathname;
|
||||
if (path === "/repos/acme/app") return Response.json({ default_branch: "main" });
|
||||
if (path === "/repos/acme/app/actions/workflows/explore-baseline.yml") return Response.json({ id: 77 });
|
||||
if (path === "/repos/acme/app/actions/artifacts") return Response.json({ artifacts: [] });
|
||||
if (path === "/repos/acme/app/actions/workflows/77/runs") {
|
||||
assert.equal(url.searchParams.get("head_sha"), sha);
|
||||
assert.equal(url.searchParams.get("branch"), "main");
|
||||
return Response.json({ workflow_runs: runs ?? [] });
|
||||
}
|
||||
if (path === "/repos/acme/app/actions/runs/20") {
|
||||
poll += 1;
|
||||
const failure = runFailures.shift();
|
||||
if (failure) return new Response("unavailable", { status: failure });
|
||||
return Response.json({ ...trustedRun, status: state().runStatus ?? "in_progress", conclusion: state().conclusion ?? null, ...run });
|
||||
}
|
||||
if (path === "/repos/acme/app/actions/runs/20/artifacts") {
|
||||
const name = url.searchParams.get("name");
|
||||
return Response.json({ artifacts: state().artifact && name === `explore-baseline-${sha}-iphone` ? [{ id: 500, name, expired: false, workflow_run: { id: 20 } }] : [] });
|
||||
}
|
||||
if (path === "/repos/acme/app/actions/runs/20/jobs") return Response.json({ jobs: [{ name: "targets", status: "completed", conclusion: "success" }, ...state().jobs ?? []] });
|
||||
if (path === "/repos/acme/app/actions/artifacts/500/zip") {
|
||||
assert.equal(init.redirect, "manual", "the redirect is followed by hand, without the token");
|
||||
return new Response(null, { status: 302, headers: { location: "https://blob.example/artifact.zip?sig=1" } });
|
||||
}
|
||||
throw new Error(`unexpected request ${url}`);
|
||||
};
|
||||
return { transport, requests, polls: () => poll + 1 };
|
||||
}
|
||||
|
||||
/** A clock the wait's pauses advance, so fifteen minutes of polling take no real time. */
|
||||
function clock() {
|
||||
let milliseconds = 0;
|
||||
return { now: () => milliseconds, sleep: async pause => { milliseconds += pause; }, elapsed: () => milliseconds };
|
||||
}
|
||||
|
||||
const waitOptions = (github, time, extra = {}) => ({ run: 20, sha, target: "iphone", timeoutMs: 15 * 60_000, environment, transport: github.transport,
|
||||
now: time.now, sleep: time.sleep, ...extra });
|
||||
|
||||
test("a baseline job in progress, then finished: the review reuses its artifact, downloaded and unpacked as download-artifact would", async t => {
|
||||
const root = await mkdtemp(join(tmpdir(), "explore-baseline-wait-"));
|
||||
t.after(() => rm(root, { recursive: true, force: true }));
|
||||
const zip = await zipFiles({ "bundle.json": "{\"side\":\"head\"}", "cache-metadata.json": "{}", "screenshots/home.png": "png" });
|
||||
const github = fakeGitHub({ zip, states: [{ jobs: [] }, { jobs: [job("queued")] }, { jobs: [job("in_progress")] }, { jobs: [job("in_progress")] },
|
||||
{ jobs: [job("completed", "success")], artifact: true }] });
|
||||
const time = clock(); const lines = [];
|
||||
const directory = join(root, "review", "baseline-download");
|
||||
assert.equal(await awaitPendingBaseline({ ...waitOptions(github, time, { progress: line => lines.push(line) }), directory }), directory);
|
||||
assert.deepEqual((await readdir(directory, { recursive: true })).sort(), ["bundle.json", "cache-metadata.json", "screenshots", "screenshots/home.png"]);
|
||||
assert.equal(await readFile(join(directory, "screenshots/home.png"), "utf8"), "png");
|
||||
assert.equal(github.polls(), 5);
|
||||
assert.deepEqual(lines, [
|
||||
"waiting for main's baseline run 20 of aaaaaaaa (iphone: not started, 0 min)",
|
||||
"waiting for main's baseline run 20 of aaaaaaaa (iphone: queued, 0 min)",
|
||||
"waiting for main's baseline run 20 of aaaaaaaa (iphone: in progress, 0 min)",
|
||||
"waiting for main's baseline run 20 of aaaaaaaa (iphone: in progress, 1 min)",
|
||||
"base: downloaded the baseline of aaaaaaaa that run 20 captured",
|
||||
]);
|
||||
assert.ok(github.requests.every(request => !request.startsWith("/repos/acme/app/actions/runs/") || request.startsWith("/repos/acme/app/actions/runs/20")),
|
||||
"only the run ci:find-baseline named is read");
|
||||
assert.deepEqual((await readdir(join(root, "review"))).sort(), ["baseline-download"], "the downloaded zip is removed once unpacked");
|
||||
});
|
||||
|
||||
test("this target's artifact is reused although another target's baseline job failed (#48)", async () => {
|
||||
const github = fakeGitHub({ states: [{ jobs: [job("completed", "failure", "duo-open"), job("in_progress")] },
|
||||
{ runStatus: "completed", conclusion: "failure", jobs: [job("completed", "failure", "duo-open"), job("completed", "success")], artifact: true }] });
|
||||
assert.deepEqual(await waitForBaselineArtifact(waitOptions(github, clock())), { artifact: { id: 500, run: 20 } });
|
||||
});
|
||||
|
||||
test("a baseline job that fails or is cancelled ends the wait, and the review captures the base itself", async t => {
|
||||
for (const [conclusion, reason] of [["failure", "its iphone job failed"], ["cancelled", "its iphone job was cancelled"], ["skipped", "its iphone job ended (skipped)"]]) {
|
||||
const github = fakeGitHub({ states: [{ jobs: [job("in_progress")] }, { jobs: [job("completed", conclusion)] }] });
|
||||
assert.deepEqual(await waitForBaselineArtifact(waitOptions(github, clock())), { reason }, conclusion);
|
||||
}
|
||||
const root = await mkdtemp(join(tmpdir(), "explore-baseline-wait-"));
|
||||
t.after(() => rm(root, { recursive: true, force: true }));
|
||||
const github = fakeGitHub({ states: [{ jobs: [job("in_progress")] }, { jobs: [job("completed", "failure")] }] });
|
||||
const lines = [];
|
||||
assert.equal(await awaitPendingBaseline({ ...waitOptions(github, clock(), { progress: line => lines.push(line) }), directory: join(root, "baseline-download") }), undefined);
|
||||
assert.equal(lines.at(-1), "base: stopped waiting for the baseline of aaaaaaaa (its iphone job failed); capturing the base too");
|
||||
assert.ok(!github.requests.some(request => request.includes("/zip")), "nothing is downloaded");
|
||||
await assert.rejects(stat(join(root, "baseline-download")), { code: "ENOENT" });
|
||||
});
|
||||
|
||||
test("a run that ends without this target's job, or a job whose artifact is never listed, ends the wait after a minute's grace", async () => {
|
||||
const ended = fakeGitHub({ states: [{ runStatus: "completed", conclusion: "failure", jobs: [] }] });
|
||||
const time = clock();
|
||||
assert.deepEqual(await waitForBaselineArtifact(waitOptions(ended, time)), { reason: "the run ended (failure) without a iphone baseline job" });
|
||||
assert.equal(time.elapsed(), 60_000);
|
||||
const unlisted = fakeGitHub({ states: [{ jobs: [job("completed", "success")] }] });
|
||||
assert.deepEqual(await waitForBaselineArtifact(waitOptions(unlisted, clock())), { reason: `its iphone job finished, but GitHub lists no explore-baseline-${sha}-iphone artifact` });
|
||||
});
|
||||
|
||||
test("the wait gives up at its time limit, polling every 20 s", async () => {
|
||||
const github = fakeGitHub({ states: [{ jobs: [job("in_progress")] }] });
|
||||
const time = clock(); const lines = [];
|
||||
assert.deepEqual(await waitForBaselineArtifact(waitOptions(github, time, { progress: line => lines.push(line) })), { reason: "its iphone job is still in progress after 15 min" });
|
||||
assert.equal(time.elapsed(), 15 * 60_000);
|
||||
assert.equal(github.polls(), 15 * 3 + 1);
|
||||
assert.equal(lines.length, 15, "one line a minute, not one a poll");
|
||||
assert.deepEqual(await waitForBaselineArtifact(waitOptions(fakeGitHub({ states: [{ jobs: [job("in_progress")] }] }), clock(), { timeoutMs: 50_000, pollMs: 30_000 })),
|
||||
{ reason: "its iphone job is still in progress after 1 min" }, "the last pause ends at the limit");
|
||||
});
|
||||
|
||||
test("only a baseline run of exactly the base commit, on the default branch of this repository, from a push or dispatch, is waited for", async () => {
|
||||
for (const mutation of [{ head_branch: "feature" }, { event: "pull_request_target" }, { event: "pull_request" }, { workflow_id: 99 },
|
||||
{ head_sha: "b".repeat(40) }, { head_repository: { full_name: "fork/app" } }]) {
|
||||
const github = fakeGitHub({ run: mutation, states: [{ jobs: [job("completed", "success")], artifact: true }] });
|
||||
assert.deepEqual(await waitForBaselineArtifact(waitOptions(github, clock())), { reason: "run 20 is not a baseline run of aaaaaaaa on main" }, JSON.stringify(mutation));
|
||||
assert.ok(!github.requests.some(request => request.startsWith("/repos/acme/app/actions/runs/20/")), "nothing of an untrusted run is read");
|
||||
}
|
||||
const dispatched = fakeGitHub({ run: { event: "workflow_dispatch" }, states: [{ artifact: true }] });
|
||||
assert.deepEqual(await waitForBaselineArtifact(waitOptions(dispatched, clock())), { artifact: { id: 500, run: 20 } });
|
||||
});
|
||||
|
||||
test("ci:find-baseline names a trusted baseline run still capturing the base commit, preferring the one already capturing", async () => {
|
||||
const run = (id, status, mutation = {}) => ({ ...trustedRun, id, status, conclusion: status === "completed" ? "success" : null, ...mutation });
|
||||
const lookup = runs => findPendingBaselineRun({ sha, target: "iphone", environment, transport: fakeGitHub({ runs }).transport });
|
||||
assert.deepEqual(await lookup([run(30, "completed"), run(31, "in_progress", { event: "pull_request_target" }), run(33, "queued"), run(32, "in_progress"),
|
||||
run(34, "in_progress", { head_branch: "feature" }), run(35, "in_progress", { head_repository: { full_name: "fork/app" } })]), { run: 32 });
|
||||
assert.deepEqual(await lookup([run(33, "queued"), run(36, "waiting")]), { run: 36 }, "otherwise the newest");
|
||||
assert.equal(await lookup([run(30, "completed"), run(31, "in_progress", { workflow_id: 99 })]), undefined);
|
||||
assert.deepEqual(await findOptionalBaseline({ sha, target: "iphone", environment, transport: fakeGitHub({ runs: [run(32, "in_progress")] }).transport }), { pending: { run: 32 } },
|
||||
"with no finished baseline to download");
|
||||
const outage = async input => new URL(input).pathname.endsWith("/77/runs") ? new Response("bad gateway", { status: 502 }) : fakeGitHub().transport(input, { headers: { authorization: "Bearer test-token" } });
|
||||
const failed = await findOptionalBaseline({ sha, target: "iphone", environment, transport: outage });
|
||||
assert.equal(failed.pending, undefined);
|
||||
assert.match(failed.warning.message, /^No baseline run in progress is waited for, so capture captures the base too: GitHub answered HTTP 502/);
|
||||
});
|
||||
|
||||
test("a GitHub 5xx while polling is retried, and a missing permission ends the wait without failing the review", async t => {
|
||||
const github = fakeGitHub({ runFailures: [0, 502, 503], states: [{ jobs: [job("in_progress")] }, { jobs: [job("in_progress")] }, { jobs: [job("in_progress")] }, { artifact: true }] });
|
||||
t.mock.method(process.stderr, "write", () => true);
|
||||
assert.deepEqual(await waitForBaselineArtifact(waitOptions(github, clock())), { artifact: { id: 500, run: 20 } });
|
||||
const outage = fakeGitHub({ runFailures: [502, 502, 502, 502, 502] });
|
||||
assert.match((await waitForBaselineArtifact(waitOptions(outage, clock()))).reason, /^GITHUB_REQUEST_FAILED: GitHub answered HTTP 502/, "an outage that outlasts its retries ends the wait");
|
||||
const denied = fakeGitHub({ runFailures: [403] });
|
||||
assert.deepEqual(await waitForBaselineArtifact(waitOptions(denied, clock())),
|
||||
{ reason: "GITHUB_PERMISSION_DENIED: GitHub answered HTTP 403 for repos/acme/app/actions/runs/20." });
|
||||
assert.equal(denied.polls(), 1, "a missing permission is not retried");
|
||||
const tokenless = await waitForBaselineArtifact(waitOptions(fakeGitHub(), clock(), { environment: { GITHUB_REPOSITORY: "acme/app" } }));
|
||||
assert.match(tokenless.reason, /^GITHUB_TOKEN_MISSING/);
|
||||
});
|
||||
|
||||
test("an artifact with a path outside its directory is refused before anything is written, and base captures as before", async t => {
|
||||
const root = await mkdtemp(join(tmpdir(), "explore-baseline-wait-"));
|
||||
t.after(() => rm(root, { recursive: true, force: true }));
|
||||
const github = fakeGitHub({ zip: await zipFiles({ "bundle.json": "{}", "../escape.txt": "outside" }), states: [{ artifact: true }] });
|
||||
const lines = [];
|
||||
assert.equal(await awaitPendingBaseline({ ...waitOptions(github, clock(), { progress: line => lines.push(line) }), directory: join(root, "review", "baseline-download") }), undefined);
|
||||
// zip.js refuses the entry as it reads the archive's directory, before the unpacking's own path check would.
|
||||
assert.equal(lines.at(-1), "base: the baseline of aaaaaaaa from run 20 could not be downloaded (Unsafe filename); capturing the base too");
|
||||
assert.deepEqual(await readdir(join(root, "review")), [], "neither the download nor a partial directory is left");
|
||||
await assert.rejects(stat(join(root, "escape.txt")), { code: "ENOENT" });
|
||||
});
|
||||
|
||||
test("a review that no longer needs the baseline stops waiting at once", async () => {
|
||||
const github = fakeGitHub({ states: [{ jobs: [job("in_progress")] }] });
|
||||
const controller = new AbortController();
|
||||
const waiting = waitForBaselineArtifact({ run: 20, sha, target: "iphone", timeoutMs: 15 * 60_000, environment, transport: github.transport, signal: controller.signal });
|
||||
setTimeout(() => controller.abort(), 50);
|
||||
const started = Date.now();
|
||||
assert.deepEqual(await waiting, { reason: "the review stopped waiting" });
|
||||
assert.ok(Date.now() - started < 5_000);
|
||||
});
|
||||
@@ -8,7 +8,8 @@ import { tsImport } from "tsx/esm/api";
|
||||
const { runBlackBoxExploration, screenEvidenceKey, elementsOf, fieldName, systemDialogTitle } = await tsImport("../../src/binary/explorer.ts", import.meta.url);
|
||||
const { ExploreStore, validateMap } = await tsImport("../../src/map/store.ts", import.meta.url);
|
||||
const { CONTRACT } = await tsImport("../../src/types.ts", import.meta.url);
|
||||
const { assertAppUrl, describeForeground, foreignNode, guardPositionals, leftForeground, retryWhileBusy } = await tsImport("../../src/binary/run.ts", import.meta.url);
|
||||
const tsRun = await tsImport("../../src/binary/run.ts", import.meta.url);
|
||||
const { assertAppUrl, describeForeground, foreignNode, guardPositionals, leftForeground, retryWhileBusy } = tsRun;
|
||||
|
||||
const PNG = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==", "base64");
|
||||
const SCREENS = {
|
||||
@@ -881,3 +882,328 @@ test("when nothing passes preflight, the first failure is reported", async () =>
|
||||
const createSession = (config, { auth }) => ({ async preflight() { throw new Error(`${config.provider} ${auth} refused`); } });
|
||||
await assert.rejects(preflightModelSession({ createSession }, {}, () => {}), /anthropic local-login refused/);
|
||||
});
|
||||
|
||||
test("a landscape frame on a portrait panel (an opened iPhone Duo) is stored upright", async () => {
|
||||
const { createRequire } = await import("node:module");
|
||||
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
|
||||
const raw = new Png({ width: 1, height: 2 });
|
||||
raw.data.set([255, 0, 0, 255, 0, 0, 255, 255]);
|
||||
const png = Png.sync.write(raw);
|
||||
const { map, root } = await harness([["observe", { includeImage: true }], ["capture", { frameId: "frame-0", title: "Home" }], ["finish", { summary: "One." }]], {
|
||||
observe: async includeImage => ({ nodes: SCREENS.home, capturedAt: new Date().toISOString(), ...(includeImage ? { frameId: "frame-0", png, width: 1, height: 2, landscape: true } : {}) }) });
|
||||
const artifact = map.artifacts.find(item => item.kind === "screenshot");
|
||||
assert.deepEqual([artifact.width, artifact.height], [2, 1]);
|
||||
const { readFile } = await import("node:fs/promises");
|
||||
const stored = Png.sync.read(await readFile(join(root, artifact.path)));
|
||||
// The panel's top (red) is the UI's right-hand side once turned a quarter clockwise.
|
||||
assert.deepEqual([...stored.data], [0, 0, 255, 255, 255, 0, 0, 255]);
|
||||
});
|
||||
|
||||
/** A PNG of `width` × `height` RGBA pixels, each given as [r, g, b, a] row by row. */
|
||||
async function pngOf(width, height, pixels) {
|
||||
const { createRequire } = await import("node:module");
|
||||
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
|
||||
const image = new Png({ width, height });
|
||||
image.data.set(pixels.flat());
|
||||
return Png.sync.write(image);
|
||||
}
|
||||
|
||||
test("an opened Duo's frame shows the same pixels as a screen already seen once turned, so it is that screen (AudioScape's playlist title, rotated)", async () => {
|
||||
const tabs = [{ type: "Button", label: "Home", hittable: true }, { type: "Button", label: "Library", hittable: true }];
|
||||
const library = [{ type: "StaticText", label: "Library" }, { type: "Button", label: "My Playlist", hittable: true }, ...tabs];
|
||||
const trees = { library, titled: [{ type: "StaticText", label: "My Playlist" }, ...tabs], untitled: [...tabs] };
|
||||
const images = { library: await pngOf(1, 2, [[255, 0, 0, 255], [0, 0, 255, 255]]), playlist: await pngOf(1, 2, [[0, 255, 0, 255], [0, 0, 255, 255]]) };
|
||||
let visits = 0; let frames = 0; let where = "library";
|
||||
const { map } = await harness([
|
||||
["observe", { includeImage: true }], ["capture", { frameId: "frame-0", title: "Library" }],
|
||||
["act", { action: "tap", label: "My Playlist" }], ["observe", { includeImage: true }],
|
||||
["act", { action: "back" }], ["observe", { includeImage: true }],
|
||||
["act", { action: "tap", label: "My Playlist" }], ["observe", { includeImage: true }],
|
||||
["finish", { summary: "Done." }],
|
||||
], { observe: async includeImage => ({ nodes: trees[where], capturedAt: new Date().toISOString(),
|
||||
...(includeImage ? { frameId: `frame-${frames++}`, png: Buffer.from(images[where === "library" ? "library" : "playlist"]), width: 1, height: 2, landscape: true } : {}) }),
|
||||
act: async action => { if (action.action === "back") where = "library"; else where = visits++ === 0 ? "titled" : "untitled"; } });
|
||||
assert.equal(map.screens.length, 2, "the second read of the playlist did not mint a screen");
|
||||
assert.equal(map.observations.filter(item => item.screenId === map.screens[1].id).length, 2, "both reads are observations of the one playlist screen");
|
||||
assert.equal(map.artifacts.filter(item => item.kind === "screenshot").length, 2);
|
||||
});
|
||||
|
||||
test("a coordinate tap on an opened Duo is named by where it shows in the upright screenshot", async () => {
|
||||
// The inner panel's portrait axes: a button near the portrait top right is at the upright bottom right.
|
||||
const home = [
|
||||
{ type: "Application", label: "", rect: { x: 0, y: 0, width: 669, height: 951 } },
|
||||
{ type: "Button", label: "", hittable: true, rect: { x: 600, y: 40, width: 44, height: 44 } },
|
||||
];
|
||||
let frames = 0; let tapped = false;
|
||||
// The look without an image carries no landscape flag; the screen's stored screenshot still says it was turned.
|
||||
const { map } = await harness([
|
||||
["observe", { includeImage: true }], ["capture", { frameId: "frame-0", title: "Home" }], ["observe", { includeImage: false }],
|
||||
["act", { action: "tap", frameId: "frame-0", x: 622 / 669, y: 62 / 951 }],
|
||||
["finish", { summary: "Done." }],
|
||||
], { observe: async includeImage => ({ nodes: tapped ? SCREENS.settings : home, capturedAt: new Date().toISOString(),
|
||||
...(includeImage ? { frameId: `frame-${frames++}`, png: await pngOf(1, 2, [[255, 0, 0, 255], [0, 0, 255, 255]]), width: 1, height: 2, landscape: true } : {}) }),
|
||||
act: async () => { tapped = true; } });
|
||||
assert.equal(map.actionAttempts.find(item => item.action.kind === "press").action.label, "icon button, bottom right");
|
||||
assert.deepEqual(map.artifacts.find(item => item.kind === "screenshot").turned, { width: 669, height: 951 });
|
||||
});
|
||||
|
||||
test("turnedClockwise puts the portrait top on the right and keeps the colour chunks", async () => {
|
||||
const { turnedClockwise, assertDecodedPng } = await tsImport("../../src/shared/png-evidence.ts", import.meta.url);
|
||||
const { createRequire } = await import("node:module");
|
||||
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
|
||||
const { crc32 } = await import("node:zlib");
|
||||
// 3 wide and 2 tall; each pixel's red is its index, so where it lands says where it came from.
|
||||
const plain = await pngOf(3, 2, [0, 1, 2, 3, 4, 5].map(index => [index * 40, 0, 0, 255]));
|
||||
const chunk = (type, data) => {
|
||||
const body = Buffer.concat([Buffer.from(type, "latin1"), data]), out = Buffer.alloc(body.length + 8);
|
||||
out.writeUInt32BE(data.length, 0); body.copy(out, 4); out.writeUInt32BE(crc32(body), body.length + 4);
|
||||
return out;
|
||||
};
|
||||
const gamma = Buffer.alloc(4); gamma.writeUInt32BE(45455);
|
||||
const source = Buffer.concat([plain.subarray(0, 33), chunk("sRGB", Buffer.from([0])), chunk("gAMA", gamma), plain.subarray(33)]);
|
||||
const turned = turnedClockwise(source);
|
||||
assertDecodedPng(turned, "Turned");
|
||||
const decoded = Png.sync.read(turned);
|
||||
assert.deepEqual([decoded.width, decoded.height], [2, 3]);
|
||||
// Source (x, y) lands at (height - 1 - y, x): the top row (0, 1, 2) runs down the right-hand column.
|
||||
const red = [...Array(6).keys()].map(index => decoded.data[index * 4] / 40);
|
||||
assert.deepEqual(red, [3, 0, 4, 1, 5, 2]);
|
||||
const types = [];
|
||||
for (let offset = 8; offset < turned.length; offset += 12 + turned.readUInt32BE(offset)) types.push(turned.toString("latin1", offset + 4, offset + 8));
|
||||
assert.deepEqual(types.slice(0, 3), ["IHDR", "sRGB", "gAMA"]);
|
||||
assert.ok(turned.includes(gamma), "the gamma is the source's");
|
||||
});
|
||||
|
||||
test("only an opened Duo's frame on its inner panel's portrait axes is turned upright", () => {
|
||||
const { duoScreenshotTurn } = tsRun;
|
||||
const root = (width, height) => ({ type: "Application", label: "Expo Go", rect: { x: 0, y: 0, width, height } });
|
||||
const frame = (deviceModel, nodes, width, height, platform = "ios") => duoScreenshotTurn({ platform, deviceModel, nodes, width, height });
|
||||
const cases = [
|
||||
["Duo open", frame("iPhone Duo", [root(669, 951)], 2007, 2853), "turn"],
|
||||
["Duo half-open", frame("iPhone Duo", [root(669, 951), { type: "Button", label: "Home", rect: { x: 20, y: 40, width: 44, height: 44 } }], 2007, 2853), "turn"],
|
||||
["Duo closed", frame("iPhone Duo", [root(466, 678)], 1398, 2034), "keep"],
|
||||
["iPhone 17 Pro", frame("iPhone 17 Pro", [root(402, 874)], 1206, 2622), "keep"],
|
||||
["an iPhone in landscape", frame("iPhone 17 Pro", [root(874, 402)], 2622, 1206), "keep"],
|
||||
["iPad", frame("iPad Pro 13-inch (M5)", [root(1032, 1376)], 2064, 2752), "keep"],
|
||||
["Android", frame("Pixel 9", [root(669, 951)], 2007, 2853, "android"), "keep"],
|
||||
["an empty tree", frame("iPhone Duo", [], 2007, 2853), "keep"],
|
||||
["a first node that isn't the Application", frame("iPhone Duo", [{ type: "Window", rect: { x: 0, y: 0, width: 300, height: 200 } }, root(669, 951)], 2007, 2853), "turn"],
|
||||
["a root off by a fraction of a point", frame("iPhone Duo", [root(669.33, 951)], 2008, 2853), "turn"],
|
||||
["an already-landscape image", frame("iPhone Duo", [root(669, 951)], 2853, 2007), "keep"],
|
||||
["the inner panel on its own axes", frame("iPhone Duo", [root(951, 669)], 2853, 2007), "keep"],
|
||||
["a Duo root on neither panel", frame("iPhone Duo", [root(700, 1000)], 2100, 3000), "unmatched"],
|
||||
["a Duo root the image does not show", frame("iPhone Duo", [root(669, 951)], 2000, 2000), "unmatched"],
|
||||
];
|
||||
for (const [name, actual, expected] of cases) assert.equal(actual, expected, name);
|
||||
});
|
||||
|
||||
test("the agent API lists a turned screenshot's element bounds on its upright axes", async () => {
|
||||
const { agentApiReply } = await tsImport("../../src/report/agent-api.ts", import.meta.url);
|
||||
const sha = "a".repeat(64);
|
||||
const map = { screens: [{ id: "scr_home", title: "Home", observationIds: ["obs_1"] }], observations: [{ id: "obs_1", screenshotArtifactId: "art_1" }],
|
||||
artifacts: [{ id: "art_1", kind: "screenshot", sha256: sha, width: 2853, height: 2007, turned: { width: 669, height: 951 } }],
|
||||
elements: [{ id: "el_1", screenId: "scr_home", label: "Settings", role: "button", bounds: { x: 600, y: 40, width: 44, height: 30, unit: "points" } }], transitions: [], variants: [], gaps: [] };
|
||||
const loaded = { kind: "exploration", file: "exploration.json", raw: new Uint8Array(), json: { map }, entries: new Map(), read: async () => undefined };
|
||||
const reply = await agentApiReply({ pageUrl: "https://example.test/p/token", platforms: ["ios"], expiresAt: null }, loaded, "/screens/scr_home", new URLSearchParams(), "application/json");
|
||||
const body = JSON.parse(await new Response(reply.body).text());
|
||||
// Portrait (600, 40) on a 951-point-tall screen shows at (951 - 40 - 30, 600) upright, near the bottom right.
|
||||
assert.deepEqual(body.elements[0].bounds, { x: 881, y: 600, width: 30, height: 44, unit: "points" });
|
||||
});
|
||||
|
||||
const { assertReplayFits, createReplaySession, parseReplayPlan, replaySetup } = await tsImport("../../src/binary/replay.ts", import.meta.url);
|
||||
|
||||
function replayDevice(screens = SCREENS) {
|
||||
let screen = "home", frames = 0;
|
||||
const acted = [];
|
||||
return { acted, options: {
|
||||
observe: async includeImage => ({ nodes: screens[screen], capturedAt: new Date().toISOString(),
|
||||
...(includeImage ? { frameId: `replay-${++frames}`, png: PNG, imageSha256: `${frames}`.padStart(64, "b") } : {}) }),
|
||||
act: async action => {
|
||||
acted.push(action);
|
||||
if (action.label === "Missing" || action.label === "Nowhere") throw new Error(`No element labelled ${action.label}`);
|
||||
if (action.action === "tap" && action.label === "Settings") screen = "settings";
|
||||
if (action.action === "tap" && action.label === "Dark mode" && screens.settingsDark) screen = "settingsDark";
|
||||
if (action.action === "back") screen = "home";
|
||||
},
|
||||
} };
|
||||
}
|
||||
|
||||
test("a replay plan takes the same steps on each run: platform-only steps are skipped, optional failures are passed over and listed", async () => {
|
||||
const plan = parseReplayPlan({ steps: [
|
||||
{ capture: "Home" },
|
||||
{ action: "tap", label: "Missing", optional: true },
|
||||
{ action: "tap", label: "Android only", platforms: ["android"] },
|
||||
{ action: "tap", label: "Settings" }, { capture: "Settings" },
|
||||
{ action: "tap", label: "Dark mode" }, { capture: "Settings", state: "dark" },
|
||||
{ action: "back" },
|
||||
] });
|
||||
const device = replayDevice();
|
||||
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
|
||||
assert.equal(result.stopReason, "finished");
|
||||
assert.match(result.summary, /Every planned screen was captured\. Optional steps skipped because they failed: step 2 \(tap "Missing"\)\.$/);
|
||||
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Settings"]);
|
||||
assert.ok(!device.acted.some(action => action.label === "Android only"));
|
||||
assert.ok(map.variants.some(variant => variant.name === "dark"), "the dark state is recorded as a variant");
|
||||
});
|
||||
|
||||
test("a failed replay step stops the run, keeps what was captured, and names the screens it did not reach", async () => {
|
||||
const plan = parseReplayPlan({ steps: [
|
||||
{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" }, { action: "back" },
|
||||
{ action: "tap", label: "Nowhere" }, { capture: "Never" }, { capture: "Android screen", platforms: ["android"] },
|
||||
] });
|
||||
const device = replayDevice();
|
||||
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
|
||||
assert.equal(result.stopReason, "blocked");
|
||||
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Settings"]);
|
||||
assert.match(result.summary, /Replay step 5 \(tap "Nowhere"\) failed: .*No element labelled Nowhere.*Not captured: Never$/);
|
||||
});
|
||||
|
||||
test("a blocked step names the screens still ahead before earlier misses, within stop_blocked's limit", async () => {
|
||||
const plan = parseReplayPlan({ steps: [
|
||||
{ capture: "Home" },
|
||||
...Array.from({ length: 20 }, (_, index) => ({ capture: `Earlier miss ${index} ${"x".repeat(60)}` })),
|
||||
{ action: "tap", label: "Nowhere" },
|
||||
{ capture: "Ahead" },
|
||||
] });
|
||||
const device = replayDevice();
|
||||
const { result } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options, maxTurns: 100 });
|
||||
assert.equal(result.stopReason, "blocked");
|
||||
assert.ok(result.summary.length <= 1000, `${result.summary.length} characters`);
|
||||
assert.match(result.summary, /Not captured: Ahead, Earlier miss 0 .* and \d+ more$/);
|
||||
});
|
||||
|
||||
test("a destructive or app-leaving tap in a replay plan is refused, as a model's would be", async () => {
|
||||
const plan = parseReplayPlan({ steps: [
|
||||
{ capture: "Home" }, { action: "tap", label: "Share", optional: true }, { action: "tap", label: "Delete account" }, { capture: "Gone" },
|
||||
] });
|
||||
const device = replayDevice();
|
||||
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
|
||||
assert.equal(result.stopReason, "blocked");
|
||||
assert.match(result.summary, /^Replay step 3 \(tap "Delete account"\) failed: "Delete account" looks destructive.*Not captured: Gone$/);
|
||||
assert.deepEqual(device.acted, [], "neither tap reaches the device");
|
||||
assert.deepEqual(map.actionAttempts.map(attempt => [attempt.action.label, attempt.outcome]), [["Share", "blocked"], ["Delete account", "blocked"]]);
|
||||
});
|
||||
|
||||
test("a replay plan is checked before any device starts: unknown, hostile and malformed fields are refused", () => {
|
||||
const refused = (step, pattern) => assert.throws(() => parseReplayPlan({ steps: [step] }), error => error.code === "REPLAY_PLAN_INVALID" && pattern.test(error.message), JSON.stringify(step));
|
||||
assert.throws(() => parseReplayPlan({ steps: [] }), /no steps/);
|
||||
assert.throws(() => parseReplayPlan({ steps: [{ capture: "Home" }], step: [] }), /field "step" it does not take/);
|
||||
refused({ action: "tap" }, /exactly one of label or identifier/);
|
||||
refused({ action: "swipe" }, /needs a capture title or an action/);
|
||||
refused({ action: "constructor" }, /needs a capture title or an action/);
|
||||
refused({ capture: "Home", platforms: ["web"] }, /platforms must list ios and\/or android/);
|
||||
// A coordinate tap's fields would turn a label tap into a tap whose risk comes from `target`.
|
||||
refused({ action: "tap", label: "Delete account", frameId: "frame-1", x: 0.2, y: 0.15, target: "Next" }, /^Step 1 \(tap\) has a field "frameId" it does not take; a tap step takes action, label, identifier, platforms, optional\.$/);
|
||||
refused({ action: "tap", label: "Next", platform: ["ios"] }, /field "platform"/);
|
||||
refused({ action: "tap", label: "Next", optinal: true }, /field "optinal"/);
|
||||
refused({ capture: "Home", optional: true }, /\(capture\) has a field "optional"/);
|
||||
refused({ capture: "Home", action: "tap", label: "Next" }, /\(capture\) has a field "action"/);
|
||||
refused({ action: "back", label: "Back" }, /\(back\) has a field "label"/);
|
||||
refused({ action: "scroll", direction: ["up"] }, /scroll needs a direction/);
|
||||
refused({ action: "scroll", direction: "sideways" }, /scroll needs a direction/);
|
||||
refused({ action: "tap", label: "x".repeat(257) }, /label must be 1–256 characters without control characters/);
|
||||
refused({ action: "tap", identifier: "row\nnext" }, /identifier must be 1–256 characters/);
|
||||
refused({ action: "fill", label: "Name", text: "two\nlines" }, /fill needs text: one line/);
|
||||
refused({ action: "type", text: "x".repeat(2001) }, /type needs text/);
|
||||
refused({ action: "type", text: "{{secret:lowercase}}" }, /invalid secret reference/);
|
||||
refused({ action: "type", text: "{{secret:OPENAI_API_KEY}}" }, /cannot reference provider/);
|
||||
refused({ action: "wait", milliseconds: 15_001 }, /between 1 and 15000/);
|
||||
assert.deepEqual(parseReplayPlan({ steps: [{ capture: "Home" }] }).steps, [{ capture: "Home" }]);
|
||||
assert.deepEqual(parseReplayPlan({ steps: [{ action: "scroll", direction: "down", optional: true, platforms: ["ios"] }] }).steps,
|
||||
[{ action: "scroll", direction: "down", platforms: ["ios"], optional: true }]);
|
||||
});
|
||||
|
||||
test("a replay plan with no steps for the platform, or more than the budgets hold, is refused before any device starts", () => {
|
||||
const limits = { maxScreens: 40, maxActions: 150, maxTurns: 400 };
|
||||
assert.throws(() => assertReplayFits(parseReplayPlan({ steps: [{ capture: "Home", platforms: ["android"] }, { action: "back", platforms: ["android"] }] }), "ios", limits),
|
||||
error => error.code === "REPLAY_PLAN_EMPTY" && /No step of the replay plan runs on ios/.test(error.message));
|
||||
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" }, { action: "back" }, { action: "wait", milliseconds: 100 }] });
|
||||
assert.doesNotThrow(() => assertReplayFits(plan, "ios", limits));
|
||||
assert.throws(() => assertReplayFits(plan, "ios", { ...limits, maxScreens: 2, maxActions: 1 }),
|
||||
error => error.code === "REPLAY_PLAN_OVER_BUDGET" && /2 captures, which need a screen budget of at least 3 \(it is 2\) and 2 actions, more than the action budget of 1/.test(error.message)
|
||||
&& /--max-screens 3 --max-actions 2/.test(error.nextAction));
|
||||
assert.throws(() => assertReplayFits(plan, "ios", { ...limits, maxTurns: 8 }), /takes 9 tool calls, more than the 8/);
|
||||
});
|
||||
|
||||
test("a replay that runs out of budget names the planned screens it did not reach in the gap", async () => {
|
||||
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" }, { action: "back" }, { capture: "Later" }] });
|
||||
const device = replayDevice();
|
||||
const session = createReplaySession(plan, "ios");
|
||||
const { result, map } = await harness([], { session, budgetNote: session.budgetNote, ...device.options, maxScreens: 2 });
|
||||
assert.equal(result.stopReason, "budget-screens");
|
||||
assert.match(map.gaps.at(-1).message, /^The exploration stopped at its screens budget with \d+ controls unexplored\. Not captured: Settings, Later\.$/);
|
||||
});
|
||||
|
||||
test("a capture of a screen the plan already captured under another title is reported and does not rename it", async () => {
|
||||
const plan = parseReplayPlan({ steps: [
|
||||
{ capture: "Home" }, { capture: "Start" }, { capture: "home" }, { action: "tap", label: "Settings" }, { action: "back" },
|
||||
] });
|
||||
const device = replayDevice();
|
||||
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
|
||||
assert.equal(result.stopReason, "finished");
|
||||
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Home → Settings"]);
|
||||
assert.match(result.summary, /Not captured: Start \(looked the same as Home\)\. Also reached, not in the plan: 1 unnamed screen\.$/);
|
||||
});
|
||||
|
||||
test("a planned state that the explorer merges into an earlier screen is neither reported as unplanned nor left in the map", async () => {
|
||||
// The README's own example: Dark mode changes the tree, and the capture titled Settings folds that new screen into Settings.
|
||||
const screens = { ...SCREENS, settingsDark: [...SCREENS.settings, { type: "Button", label: "Theme options" }] };
|
||||
const plan = parseReplayPlan({ steps: [
|
||||
{ capture: "Home" }, { action: "tap", label: "Settings" }, { capture: "Settings" },
|
||||
{ action: "tap", label: "Dark mode" }, { capture: "Settings", state: "dark" },
|
||||
] });
|
||||
const device = replayDevice(screens);
|
||||
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options });
|
||||
assert.equal(result.stopReason, "finished");
|
||||
assert.deepEqual(map.screens.map(screen => screen.title), ["Home", "Settings"]);
|
||||
const settings = map.screens.find(screen => screen.title === "Settings");
|
||||
assert.ok(map.variants.some(variant => variant.screenId === settings.id && variant.name === "dark"));
|
||||
assert.equal(result.summary, "Replayed 5 planned steps on ios. Every planned screen was captured.");
|
||||
});
|
||||
|
||||
test("a long replay summary is kept within finish's limit, the most useful names first", async () => {
|
||||
const titles = Array.from({ length: 60 }, (_, index) => `Planned screen ${String(index).padStart(2, "0")} ${"y".repeat(60)}`);
|
||||
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, ...titles.map(title => ({ capture: title }))] });
|
||||
const device = replayDevice();
|
||||
const { result } = await harness([], { session: createReplaySession(plan, "ios"), ...device.options, maxTurns: 200 });
|
||||
assert.equal(result.stopReason, "finished");
|
||||
assert.ok(result.summary.length <= 4000, `${result.summary.length} characters`);
|
||||
assert.match(result.summary, /^Replayed 61 planned steps on ios\. Not captured: Planned screen 00 y+ \(looked the same as Home\), .* and \d+ more\.$/);
|
||||
});
|
||||
|
||||
test("a refused finish ends the replay with stop_blocked instead of a turn budget, and a refused stop is an error", async () => {
|
||||
const plan = parseReplayPlan({ steps: [{ action: "back" }] });
|
||||
const calls = [];
|
||||
const request = refuse => ({ signal: new AbortController().signal, callTool: async (name, input) => {
|
||||
calls.push({ name, input });
|
||||
const value = name === "get_progress" ? { captured: [] } : refuse.includes(name) ? { ok: false, code: "AGENT_ACTION_INVALID", message: `${name} refused` } : { ok: true };
|
||||
return { content: [{ type: "text", text: JSON.stringify(value) }], ...(refuse.includes(name) ? { isError: true } : {}) };
|
||||
} });
|
||||
await createReplaySession(plan, "ios").run(request(["finish"]));
|
||||
assert.deepEqual(calls.map(call => call.name), ["act", "get_progress", "finish", "stop_blocked"]);
|
||||
assert.match(calls.at(-1).input.reason, /^The replay could not finish \(finish refused\)\. Replayed 1 planned steps on ios\./);
|
||||
await assert.rejects(createReplaySession(plan, "ios").run(request(["finish", "stop_blocked"])), { code: "REPLAY_STOP_REFUSED" });
|
||||
});
|
||||
|
||||
test("a replay plan's {{secret:NAME}} inputs are read before any device starts and typed for the plan", async () => {
|
||||
const { runBinaryExploration, runSessionExploration } = await tsImport("../../src/binary/run.ts", import.meta.url);
|
||||
const { createExplorationInputs } = await tsImport("../../src/config/exploration-inputs.ts", import.meta.url);
|
||||
const plan = parseReplayPlan({ steps: [{ capture: "Home" }, { action: "fill", label: "Password", text: "{{secret:APP_TEST_PASSWORD}}" }] });
|
||||
const output = await mkdtemp(join(tmpdir(), "explore-replay-"));
|
||||
for (const run of [runBinaryExploration, runSessionExploration]) {
|
||||
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: plan, environment: {} }),
|
||||
error => error.code === "EXPLORATION_SECRET_MISSING" && /APP_TEST_PASSWORD/.test(error.message));
|
||||
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: plan, environment: {}, provider: "openai", hints: "Look around" }),
|
||||
error => error.code === "REPLAY_OPTIONS_IGNORED" && /--provider and --hints would do nothing/.test(error.message));
|
||||
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: { steps: [{ action: "tap", label: "Next", optinal: true }] }, environment: {} }), { code: "REPLAY_PLAN_INVALID" });
|
||||
await assert.rejects(run({ archive: join(output, "missing.apk"), output, replay: parseReplayPlan({ steps: [{ action: "open", url: "https://example.com" }] }), environment: {} }),
|
||||
error => error.code === "REPLAY_PLAN_INVALID" && /^Step 1: https: URLs leave the app/.test(error.message));
|
||||
}
|
||||
const device = replayDevice();
|
||||
const inputs = createExplorationInputs({ setup: replaySetup(plan), environment: { APP_TEST_PASSWORD: "hunter2-test" } });
|
||||
const { result, map } = await harness([], { session: createReplaySession(plan, "ios"), inputs, ...device.options });
|
||||
assert.equal(result.stopReason, "finished");
|
||||
assert.deepEqual(device.acted, [{ action: "fill", label: "Password", text: "hunter2-test" }]);
|
||||
assert.equal(map.actionAttempts[0].action.input.value, "{{secret:APP_TEST_PASSWORD}}");
|
||||
assert.ok(!JSON.stringify(map).includes("hunter2-test"));
|
||||
});
|
||||
@@ -29,25 +29,25 @@ async function simulatorApp(root) {
|
||||
}
|
||||
|
||||
/** simctl and agent-device on a Mac with one iPhone 17 Pro runtime; the app shows a single screen. */
|
||||
function localRunner() {
|
||||
function localRunner(device = { name: "iPhone 17 Pro", identifier: DEVICE_TYPE }, screen = { nodes: [{ type: "StaticText", label: "Welcome" }, { type: "Button", label: "Settings" }], png: PNG }) {
|
||||
const requests = [];
|
||||
const ok = data => ({ stdout: JSON.stringify({ success: true, data }), stderr: "" });
|
||||
const runner = request => {
|
||||
requests.push(request);
|
||||
const [first, second, third] = request.args;
|
||||
if (request.command === "xcrun") {
|
||||
if (second === "list" && third === "devicetypes") return { stdout: JSON.stringify({ devicetypes: [{ name: "iPhone 17 Pro", identifier: DEVICE_TYPE }] }), stderr: "" };
|
||||
if (second === "list" && third === "devicetypes") return { stdout: JSON.stringify({ devicetypes: [device] }), stderr: "" };
|
||||
if (second === "list" && third === "runtimes") return { stdout: JSON.stringify({ runtimes: [{ isAvailable: true, platform: "iOS", identifier: "com.apple.CoreSimulator.SimRuntime.iOS-27-0",
|
||||
version: "27.0", supportedDeviceTypes: [{ identifier: DEVICE_TYPE }] }] }), stderr: "" };
|
||||
version: "27.0", supportedDeviceTypes: [{ identifier: device.identifier }] }] }), stderr: "" };
|
||||
if (second === "create") return { stdout: `${UDID}\n`, stderr: "" };
|
||||
return { stdout: "", stderr: "" };
|
||||
}
|
||||
if (first === "--version") return { stdout: `${controllerVersion("agent-device")}\n`, stderr: "" };
|
||||
if (first === "open") return ok({ session: CONTROLLER_SESSION, appBundleId: "dev.example.app", device: { platform: "ios", target: "mobile", name: "Explore iPhone 17 Pro", id: UDID },
|
||||
identifiers: { session: CONTROLLER_SESSION, appBundleId: "dev.example.app", deviceName: "Explore iPhone 17 Pro", deviceId: UDID } });
|
||||
if (first === "snapshot") return ok({ nodes: [{ type: "StaticText", label: "Welcome" }, { type: "Button", label: "Settings" }] });
|
||||
if (first === "snapshot") return ok({ nodes: screen.nodes });
|
||||
if (first === "appstate") return ok({ platform: "ios", appBundleId: "dev.example.app", state: "runningForeground", surface: "app" });
|
||||
if (first === "screenshot") { writeFileSync(request.args[1], PNG); return ok({}); }
|
||||
if (first === "screenshot") { writeFileSync(request.args[1], screen.png); return ok({}); }
|
||||
return ok({});
|
||||
};
|
||||
return { requests, runner };
|
||||
@@ -85,3 +85,34 @@ test("explore:binary --simulator local runs on a simulator it creates, with no E
|
||||
assert.equal(result.session.id, `local-${UDID}`);
|
||||
assert.deepEqual(await readdir(join(root, ".cache", "expo-explore", "local-captures")), [], "nothing is left for a sweep");
|
||||
});
|
||||
|
||||
test("an opened iPhone Duo's screenshots are stored upright, and the run records the landscape they show", async t => {
|
||||
const root = await mkdtemp(join(tmpdir(), "explore-binary-duo-"));
|
||||
t.after(() => rm(root, { recursive: true, force: true }));
|
||||
const { createRequire } = await import("node:module");
|
||||
const { PNG: Png } = createRequire(import.meta.url)("pngjs");
|
||||
// The inner panel on its portrait axes at 1x, as local agent-device returns it: the tree's root is 669×951 points.
|
||||
const raw = new Png({ width: 669, height: 951 });
|
||||
for (let offset = 0; offset < raw.data.length; offset += 4) raw.data.set(offset < 669 * 4 ? [255, 0, 0, 255] : [255, 255, 255, 255], offset);
|
||||
const nodes = [{ type: "Application", label: "Example", rect: { x: 0, y: 0, width: 669, height: 951 } }, { type: "Button", label: "Settings", rect: { x: 20, y: 40, width: 80, height: 44 } }];
|
||||
const { runner } = localRunner({ name: "iPhone Duo", identifier: "com.apple.CoreSimulator.SimDeviceType.iPhone-Duo" }, { nodes, png: Png.sync.write(raw) });
|
||||
const session = { provider: "anthropic", model: "test", async preflight() {}, async run(request) {
|
||||
const observed = JSON.parse((await request.callTool("observe", { includeImage: true }, request.signal)).content[0].text);
|
||||
await request.callTool("capture", { frameId: observed.frameId, title: "Home" }, request.signal);
|
||||
await request.callTool("finish", { summary: "One." }, request.signal).catch(() => {});
|
||||
return { stopReason: "done" };
|
||||
} };
|
||||
const result = await runBinaryExploration({ archive: await simulatorApp(root), output: join(root, "out"), simulator: "local", device: "iPhone Duo", provider: "anthropic", runner,
|
||||
createSession: () => session, environment: { PATH: process.env.PATH, HOME: root, ANTHROPIC_API_KEY: "key" } });
|
||||
const { readFile } = await import("node:fs/promises");
|
||||
const map = JSON.parse(await readFile(result.mapPath, "utf8"));
|
||||
const artifact = map.artifacts.find(item => item.kind === "screenshot");
|
||||
assert.deepEqual([artifact.width, artifact.height, artifact.turned], [951, 669, { width: 669, height: 951 }]);
|
||||
const stored = Png.sync.read(await readFile(join(result.outputDirectory, artifact.path)));
|
||||
assert.deepEqual([stored.width, stored.height], [951, 669]);
|
||||
// The panel's top row (red) is the right-hand column once turned.
|
||||
assert.deepEqual([...stored.data.subarray((951 - 1) * 4, 951 * 4)], [255, 0, 0, 255]);
|
||||
assert.deepEqual([...stored.data.subarray(0, 4)], [255, 255, 255, 255]);
|
||||
assert.equal(map.run.environment.orientation, "landscape");
|
||||
assert.equal(result.runtime.orientation, "landscape");
|
||||
});
|
||||
@@ -138,6 +138,8 @@ test("capture recipes exclude display settings but bind agent instructions and c
|
||||
// How many sessions capture it does not change the evidence: a baseline captured under any ceiling serves the review.
|
||||
assert.equal(key,captureRecipeConfig({...config,capture:{...config.capture,maxSessions:16}}));
|
||||
assert.equal(captureRecipeConfig({...config,capture:{...config.capture,maxSessions:4}}),captureRecipeConfig({...config,capture:{...config.capture,maxSessions:16}}));
|
||||
// Nor does how long a review waits for a baseline still being captured.
|
||||
assert.equal(key,captureRecipeConfig({...config,capture:{...config.capture,baselineWaitMinutes:0}}));
|
||||
// An unset repack is keyed as it resolves, so configs that never chose one keep the recipe of the old written-out default.
|
||||
assert.equal(key,captureRecipeConfig({...config,capture:{...config.capture,repack:"eas"}}));
|
||||
assert.equal(JSON.parse(key).capture.repack, "eas");
|
||||
|
||||
+117
-1
@@ -69,6 +69,18 @@ const unreadableSurface = () => json({ success: false, error: { code: 'COMMAND_F
|
||||
details: { hint: 'com.apple.SafariViewService hosts the surface presented over the app, and this capture targeted that surface. No snapshot backend could read this screen (XCTest-backed snapshot tiers were deferred after recent slow accessibility work on this screen).' } } });
|
||||
|
||||
|
||||
/**
|
||||
* The iOS 26 keyboard as agent-device reports it: the Keyboard element with its keys beneath it, and the dock's Emoji and
|
||||
* Dictate buttons, which its host bridge names by their view class (brentvatne/euxy#172, review run 39362642343).
|
||||
*/
|
||||
const softKeyboard = [
|
||||
{ index: 900, type: 'Application', rect: { x: 0, y: 0, width: 402, height: 874 }, visible: true },
|
||||
{ index: 902, parentIndex: 900, type: 'Keyboard', rect: { x: 0, y: 583, width: 402, height: 233 }, visible: true },
|
||||
{ index: 903, parentIndex: 902, type: 'Key', label: 'Q', rect: { x: 5, y: 590, width: 39, height: 54 }, visible: true },
|
||||
{ index: 905, parentIndex: 900, type: 'Button', role: 'UIKeyboardDockItemButton', label: 'Emoji', rect: { x: 8, y: 806, width: 68, height: 68 }, visible: true },
|
||||
{ index: 906, parentIndex: 900, type: 'Button', role: 'UIKeyboardDockItemButton', label: 'Dictate', identifier: 'dictation', rect: { x: 325, y: 806, width: 68, height: 68 }, visible: true },
|
||||
];
|
||||
|
||||
function agentFixture(actions, observedRequests = []) {
|
||||
return { instructions: 'Capture each selected route in its normal loaded state.', contexts: {},
|
||||
provider: { provider: 'anthropic', model: 'test-model', preflight: async () => {}, run: async request => {
|
||||
@@ -117,6 +129,10 @@ async function fixture(t, settings = {}) {
|
||||
let enteredName = false; let scrolledToContinue = false; let finishHidden = false; let wedgeCleared = false; let deviceLost = false; let relaunchedAfterForeign = false; let serverStarts = 0; let unreadableLeft = settings.unreadableAfterFold ?? 0;
|
||||
// startsErrored / startsUnnamed: how many session starts EAS fails, after naming the session (it ERRORS) or before.
|
||||
let starts = 0; let currentSession = 'session-1'; const erroredSessions = new Set(); let foldUnverifiedLeft = settings.foldUnverified ?? 0;
|
||||
// listFailures: the first so many `simulator:control --list` calls fail, as a host that drops the connection does.
|
||||
let listFailuresLeft = settings.listFailures ?? 0;
|
||||
// focusedField: the screen at `path` focuses its text field, and the keyboard is up from the `keyboardFrom`th read of it.
|
||||
let focusedReads = 0;
|
||||
// blackAfterFold: the first so many folds of an EAS agent-device session light the panel but draw nothing.
|
||||
let folds = 0;
|
||||
const deviceName = settings.android ? 'Pixel 7' : 'iPhone 17 Pro';
|
||||
@@ -193,7 +209,10 @@ async function fixture(t, settings = {}) {
|
||||
// controlActions: what the session's serve-sim offers through simulator:control (none unless a test names them).
|
||||
if (command === 'simulator:control') {
|
||||
assert.equal(value(request, '--id'), currentSession, 'simulator:control names the session');
|
||||
if (request.args.includes('--list')) return json(Object.fromEntries((settings.controlActions ?? []).map(action => [action, { type: 'object' }])));
|
||||
if (request.args.includes('--list')) {
|
||||
if (listFailuresLeft > 0) { listFailuresLeft -= 1; return { stdout: '', stderr: 'The simulator preview server closed the connection before it answered.\nError: simulator:control command failed.\n' }; }
|
||||
return json(Object.fromEntries((settings.controlActions ?? []).map(action => [action, { type: 'object' }])));
|
||||
}
|
||||
assert.ok((settings.controlActions ?? []).includes(request.args[1]), `${request.args[1]} is offered by the session`);
|
||||
if (request.args[1] === 'appearance.set') appearance = request.args.at(-1);
|
||||
return json({ id: currentSession, action: request.args[1], output: '' });
|
||||
@@ -369,6 +388,8 @@ async function fixture(t, settings = {}) {
|
||||
] : []),
|
||||
...(settings.statefulSetup && current === '/transient' ? [{ type: 'Button', label: 'Finish', ref: 'e103', enabled: true, visible: !finishHidden }] : []),
|
||||
...(alert ? [{ type: 'Alert', label: alert.message }, ...alert.items.map((label, index) => ({ type: 'Button', label, ref: `e${index + 1}`, enabled: true, visible: true }))] : []),
|
||||
...(settings.focusedField?.path === current ? [{ type: 'TextField', label: 'Pattern name', focused: true, visible: true, rect: { x: 23, y: 386, width: 355, height: 50 } },
|
||||
...(++focusedReads >= (settings.focusedField.keyboardFrom ?? Infinity) ? softKeyboard : [])] : []),
|
||||
...(settings.missingObserver || settings.missingObserverAt === current ? [] : [observer]),
|
||||
// A second active destination is ambiguous; the host AX bridge's identical copy is not.
|
||||
...(settings.duplicateObserver ? [{ ...observer, label: JSON.stringify({ provider: 'expo-router', pathname: '/other', segments: ['other'], parameters: {} }) }] : []),
|
||||
@@ -576,6 +597,61 @@ test('when a fold on EAS leaves Argent reading the dark panel, the capture resta
|
||||
}
|
||||
});
|
||||
|
||||
// In brentvatne/euxy's main baseline run 39487618529, one failed --list made the dark session look as if it offered no
|
||||
// actions: it dropped dark, approved no link scheme, skipped the SpringBoard restart its fold needed, and failed the target.
|
||||
test('a --list that fails once still gives an Argent dark session its appearance, its link approval and its fold recovery', async t => {
|
||||
const f = await fixture(t, { argent: true, userInterfaceStyle: 'automatic', devicePose: 'open', foldUnverified: 1, listFailures: 1,
|
||||
controlActions: ['appearance.set', 'app.approveUrlScheme', 'springboard.restart'] });
|
||||
const lines = [];
|
||||
const result = await runCiMap({ ...f.options, progress: line => lines.push(line), appearancePass: { all: ['light', 'dark'], capture: ['dark'] } });
|
||||
assert.equal(result.runStatus, 'complete');
|
||||
const control = f.requests.filter(request => request.args[0] === 'simulator:control');
|
||||
assert.equal(control.filter(request => request.args.includes('--list')).length, 2);
|
||||
assert.equal(lines.some(line => /Argent cannot switch appearance/.test(line)), false, lines.join('\n'));
|
||||
assert.deepEqual(control.filter(request => request.args[1] === 'appearance.set').map(request => request.args.at(-1)), ['dark']);
|
||||
assert.equal(control.filter(request => request.args[1] === 'app.approveUrlScheme').length, 1);
|
||||
assert.equal(control.filter(request => request.args[1] === 'springboard.restart').length, 1);
|
||||
const bundle = JSON.parse(await readFile(join(f.options.outputDirectory, 'bundle.json'), 'utf8'));
|
||||
assert.deepEqual(Object.keys(bundle.images).sort(), [`${f.ids.home}@dark`, `${f.ids.settings}@dark`]);
|
||||
});
|
||||
|
||||
test('a --list that keeps failing fails the session with a retryable error before the app is installed, and stops the session', async t => {
|
||||
const f = await fixture(t, { argent: true, devicePose: 'open', listFailures: Infinity, controlActions: ['springboard.restart'] });
|
||||
await assert.rejects(runCiMap(f.options), error => error.code === 'SIMULATOR_CONTROL_UNAVAILABLE' && error.retryable === true);
|
||||
assert.equal(f.requests.filter(request => request.args[0] === 'simulator:control' && request.args.includes('--list')).length, 3);
|
||||
assert.equal(f.requests.some(request => request.command.endsWith('/argent-controller') && ['install', 'fold'].includes(request.args[0])), false);
|
||||
assert.equal(f.operations().filter(operation => operation === 'simulator:stop').length, 1);
|
||||
});
|
||||
|
||||
test('a session given only dark that cannot switch to it stops before installing or folding, and the merge keeps the light session\'s capture', async t => {
|
||||
const dark = await fixture(t, { argent: true, userInterfaceStyle: 'automatic', devicePose: 'open' });
|
||||
const lines = [];
|
||||
await runCiMap({ ...dark.options, progress: line => lines.push(line), appearancePass: { all: ['light', 'dark'], capture: ['dark'] } });
|
||||
assert.equal(dark.requests.some(request => request.command.endsWith('/argent-controller') && ['install', 'fold'].includes(request.args[0])), false);
|
||||
assert.equal(dark.operations().filter(operation => operation === 'simulator:stop').length, 1);
|
||||
assert.ok(lines.some(line => /session session-1 cannot switch to dark, the appearance it was given; it stops without capturing/.test(line)), lines.join('\n'));
|
||||
const idle = JSON.parse(await readFile(join(dark.options.outputDirectory, 'bundle.json'), 'utf8'));
|
||||
assert.deepEqual(Object.keys(idle.images), []);
|
||||
assert.equal(idle.outcomes[`${dark.ids.home}@dark`].error.code, 'SESSION_APPEARANCE_UNAVAILABLE');
|
||||
// A light session that switched appearance expects dark: each route's dark frame is reported as not captured.
|
||||
// One that could not switch either captured light alone, as one session would have: the dark session adds nothing.
|
||||
for (const [actions, expected] of [[['appearance.set'], 'failed'], [[], undefined]]) {
|
||||
const light = await fixture(t, { argent: true, userInterfaceStyle: 'automatic', devicePose: 'open', controlActions: actions });
|
||||
await runCiMap({ ...light.options, appearancePass: { all: ['light', 'dark'], capture: ['light'] } });
|
||||
await mergeCaptureExports(light.options.outputDirectory, [{ directory: dark.options.outputDirectory, scheme: 'dark' }]);
|
||||
const merged = JSON.parse(await readFile(join(light.options.outputDirectory, 'bundle.json'), 'utf8'));
|
||||
for (const id of Object.values(light.ids)) {
|
||||
assert.equal(merged.outcomes[id].status, 'captured', id);
|
||||
assert.equal(merged.outcomes[`${id}@dark`]?.status, expected, `${id}@dark with ${actions.join(', ') || 'no actions'}`);
|
||||
}
|
||||
}
|
||||
// A session that cannot take the side's primary appearance fails it: the merge has nothing to add the others to.
|
||||
const primary = await fixture(t, { argent: true, devicePose: 'open' });
|
||||
await assert.rejects(runCiMap({ ...primary.options, appearance: ['dark', 'light'], appearancePass: { all: ['dark', 'light'], capture: ['dark'] } }),
|
||||
error => error.code === 'SESSION_APPEARANCE_UNAVAILABLE');
|
||||
assert.equal(primary.requests.some(request => request.command.endsWith('/argent-controller') && request.args[0] === 'fold'), false);
|
||||
});
|
||||
|
||||
test('a session EAS ends while starting is replaced by a new one, under a name of its own, up to five times with a growing wait', async t => {
|
||||
const f = await fixture(t, { startsErrored: 1 });
|
||||
const lines = []; const waits = [];
|
||||
@@ -1181,6 +1257,46 @@ test('every app link settles before the screen is read', async t => {
|
||||
assert.ok(opened >= 0 && settled > opened && settled < read, 'the settle runs between the link and the first read');
|
||||
});
|
||||
|
||||
const keyboardLog = async f => (await readFile(join(f.options.mapDirectory, 'capture-diagnostics', 'keyboard.jsonl'), 'utf8'))
|
||||
.trim().split('\n').map(line => JSON.parse(line));
|
||||
|
||||
test('a screen whose text field has the focus is captured once its keyboard is up and still, and the keyboard is not one of its elements', async t => {
|
||||
// euxy#172 (review run 39362642343): the rename sheet focuses its field, and the keyboard rises after the sheet settles.
|
||||
const f = await fixture(t, { routes: [route('rename')], focusedField: { path: '/rename', keyboardFrom: 2 } });
|
||||
const result = await runCiMap(f.options);
|
||||
assert.equal(result.outcomes[f.ids.rename].status, 'captured');
|
||||
assert.equal(result.outcomes[f.ids.rename].attempts, 1);
|
||||
assert.deepEqual((await keyboardLog(f)).map(line => [line.route, line.outcome, line.reads]), [[f.ids.rename, 'shown', 2]],
|
||||
'read until the keyboard was up in the same place twice');
|
||||
const calls = f.requests.filter(request => request.command === 'fake-controller').map(request => request.args[0]);
|
||||
const shot = calls.lastIndexOf('screenshot');
|
||||
assert.deepEqual(calls.slice(shot - 3, shot), ['snapshot', 'snapshot', 'snapshot'], 'the destination read, then the keyboard reads, then the screenshot');
|
||||
const map = await readExportedMap(f.options.outputDirectory);
|
||||
const labels = map.observations.flatMap(observation => observation.elements.map(element => element.label));
|
||||
assert.ok(labels.includes('Pattern name'));
|
||||
for (const label of ['Q', 'Emoji', 'Dictate']) assert.equal(labels.includes(label), false, `${label} is the keyboard's, not the app's`);
|
||||
});
|
||||
|
||||
test('a focused field whose keyboard never rises is captured after about a second and at least two reads, without it', async t => {
|
||||
const f = await fixture(t, { routes: [route('rename')], focusedField: { path: '/rename' } });
|
||||
const result = await runCiMap(f.options);
|
||||
assert.equal(result.outcomes[f.ids.rename].status, 'captured');
|
||||
const [line] = await keyboardLog(f);
|
||||
assert.equal(line.outcome, 'absent');
|
||||
assert.ok(line.reads >= 2 && line.reads <= 6 && line.elapsedMs >= 1000, `gave up after ${line.reads} reads in ${line.elapsedMs} ms`);
|
||||
assert.equal(f.operations().includes('keyboard'), false, 'the keyboard is never dismissed or summoned');
|
||||
// A screen without a focused field is not read again.
|
||||
const plain = await fixture(t, { routes: [route('home')] });
|
||||
await runCiMap(plain.options);
|
||||
assert.equal(existsSync(join(plain.options.mapDirectory, 'capture-diagnostics', 'keyboard.jsonl')), false);
|
||||
});
|
||||
|
||||
test('a keyboard already up when the screen is read is captured once a later read finds it in the same place', async t => {
|
||||
const f = await fixture(t, { routes: [route('rename')], focusedField: { path: '/rename', keyboardFrom: 1 } });
|
||||
assert.equal((await runCiMap(f.options)).outcomes[f.ids.rename].status, 'captured');
|
||||
assert.deepEqual((await keyboardLog(f)).map(line => [line.outcome, line.reads]), [['shown', 1]]);
|
||||
});
|
||||
|
||||
test('an application Open button is never an implicit confirmation action', async t => {
|
||||
const f = await fixture(t, { routes: [route('home')], appOpenButton: true });
|
||||
const result = await runCiMap(f.options);
|
||||
|
||||
@@ -331,3 +331,15 @@ test("compare CLI emits a comparison, review, and catalog that reload as a works
|
||||
));
|
||||
assert.deepEqual(shared.baselineScreenIds, ["shared-home"]);
|
||||
});
|
||||
|
||||
test("fingerprints from different Explore versions are not comparable: the match needs review instead of reading as changed", async () => {
|
||||
// A map saved by Explore 0.20.x hashes the tree with the keyboard and the focus in it, unprefixed; a2 hashes leave them out.
|
||||
const baseline = makeMap("main-eeeeeee", [{ id: "home", route: "/", acc: "0".repeat(64) }]);
|
||||
const current = makeMap("pr-3-fffffff", [{ id: "home", route: "/", acc: `a2:${"1".repeat(64)}` }]);
|
||||
const { comparison, review } = generateComparison(baseline, current);
|
||||
await validateComparisonDocument(comparison, baseline, current);
|
||||
const home = review.entries.find((entry) => entry.entryId === "match-home");
|
||||
assert.equal(home.needsReview, true);
|
||||
assert.match(entryById(comparison, "match-home").rationale, /different Explore versions/);
|
||||
assert.ok(home.notes.some((note) => /compare the screenshots/.test(note)));
|
||||
});
|
||||
@@ -823,3 +823,48 @@ test("a capture's snapshot backend is read from its saved controller response",
|
||||
assert.equal(snapshotBackend(directory, map("artifacts/accessibility/missing.json"), "o"), undefined);
|
||||
assert.equal(snapshotBackend(directory, map("../outside.json"), "o"), undefined);
|
||||
});
|
||||
|
||||
/**
|
||||
* Tempo on the closed iPhone Duo in brentvatne/euxy#172 (review run 39362642343): a 1398 × 2034 screenshot of a
|
||||
* 466 × 678 pt screen, and a footnote whose text changed below the fold of its scrolling sheet, so the screenshots match.
|
||||
* The published review carries no accessibility tree, so the footnote's real bounds are unknown: these tests pin the
|
||||
* rule (more than half of the changed element past the screen's edge), not where euxy's footnote was.
|
||||
*/
|
||||
function tempoSide(sideName, footnote, footnoteBounds, image = { width: 1398, height: 2034 }) {
|
||||
const evidence = side(sideName, [
|
||||
element({ elementId: "app", role: "other", label: "euxy", actions: [], actionable: false, bounds: { x: 0, y: 0, width: 466, height: 678, unit: "points" } }),
|
||||
element({ elementId: "title", role: "text", label: "Base Resolution", actions: [], actionable: false, bounds: { x: 26, y: 560, width: 300, height: 24, unit: "points" } }),
|
||||
element({ elementId: `${sideName}-footnote`, role: "text", label: footnote, actions: [], actionable: false, bounds: footnoteBounds }),
|
||||
]);
|
||||
if (image) {
|
||||
evidence.map.artifacts.push({ id: `${sideName}-shot`, kind: "screenshot", path: `${sideName}.png`, sha256: "0".repeat(64), mimeType: "image/png", ...image, createdAt: "2026-09-01T01:00:00.000Z" });
|
||||
evidence.map.observations[0].screenshotArtifactId = `${sideName}-shot`;
|
||||
}
|
||||
return evidence;
|
||||
}
|
||||
const NEW_LANES = "Changes apply live while playing. Base resolution sets the step grid for new lanes.";
|
||||
const ADD_NEXT = "Changes apply live while playing. Base resolution sets the step grid for lanes you add next.";
|
||||
// Its top 12 pt on the screen, 28 pt below the bottom edge: 70% past it.
|
||||
const mostlyBelow = { x: 26, y: 666, width: 414, height: 40, unit: "points" };
|
||||
const offScreenOf = (bounds, assessment = visual("unchanged"), image) => compareSurfaceEvidence({ impact: impact(),
|
||||
base: tempoSide("base", NEW_LANES, bounds, image), head: tempoSide("head", ADD_NEXT, bounds, image), visualAssessment: assessment }).offScreen;
|
||||
|
||||
test("matching screenshots of a change whose elements lie mostly past the screen's edge are off-screen", () => {
|
||||
assert.equal(offScreenOf(mostlyBelow), true);
|
||||
// A turned frame (an opened iPhone Duo shown upright) keeps the tree on the portrait axes: the screen is still found.
|
||||
assert.equal(offScreenOf(mostlyBelow, visual("unchanged"), { width: 2034, height: 1398 }), true);
|
||||
});
|
||||
|
||||
test("a change on the screen, partly on it, with differing screenshots, or with no screen to measure against is never off-screen", () => {
|
||||
const cases = {
|
||||
"an on-screen text change": [{ x: 26, y: 600, width: 414, height: 40, unit: "points" }],
|
||||
// Its top 28 pt on the screen, 12 pt below: mostly visible, so matching pixels prove nothing about it.
|
||||
"a footnote mostly on the screen": [{ x: 26, y: 650, width: 414, height: 40, unit: "points" }],
|
||||
// A control nudged 2 pt past the left edge, otherwise wholly visible.
|
||||
"a control just crossing an edge": [{ x: -2, y: 600, width: 300, height: 40, unit: "points" }],
|
||||
"screenshots that differ": [mostlyBelow, visual("modified")],
|
||||
"no screenshot size": [mostlyBelow, visual("unchanged"), null],
|
||||
"a changed element without bounds": [undefined],
|
||||
};
|
||||
for (const [name, args] of Object.entries(cases)) assert.equal(offScreenOf(...args), undefined, name);
|
||||
});
|
||||
@@ -47,6 +47,12 @@ test("rejects obsolete configuration and gives the exact next action", () => {
|
||||
assert.equal(validateExploreConfig({ ...base(), capture: { readinessSeconds: 12 } }).capture.readinessSeconds, 12);
|
||||
assert.equal("readinessSeconds" in validateExploreConfig(base()).capture, false);
|
||||
assert.throws(() => validateExploreConfig({ ...base(), capture: { readinessSeconds: 0 } }), /capture.readinessSeconds must be a positive integer/);
|
||||
// 0 never waits for a baseline still being captured; absent, the review waits DEFAULT_BASELINE_WAIT_MINUTES.
|
||||
for (const minutes of [0, 15, 60]) assert.equal(validateExploreConfig({ ...base(), capture: { baselineWaitMinutes: minutes } }).capture.baselineWaitMinutes, minutes);
|
||||
assert.equal("baselineWaitMinutes" in validateExploreConfig(base()).capture, false);
|
||||
for (const minutes of [-1, 61, 1.5, "15"]) {
|
||||
assert.throws(() => validateExploreConfig({ ...base(), capture: { baselineWaitMinutes: minutes } }), /capture.baselineWaitMinutes must be a whole number from 0 to 60/, String(minutes));
|
||||
}
|
||||
});
|
||||
|
||||
test("validates device, capture limits, providers, paths, and custom app schemes", () => {
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import test from "node:test";
|
||||
import { tsImport } from "tsx/esm/api";
|
||||
|
||||
const { snapshotFromTree } = await tsImport("../../src/capture/argent-driver.ts", import.meta.url);
|
||||
const { focusedTextInput, keyboardBounds, keyboardNodes, keyboardSettle } = await tsImport("../../src/capture/keyboard.ts", import.meta.url);
|
||||
const { parseControllerOutput } = await tsImport("../../src/capture/controller-output.ts", import.meta.url);
|
||||
const { normalizeSnapshot } = await tsImport("../../src/map/store.ts", import.meta.url);
|
||||
|
||||
const fixture = name => JSON.parse(readFileSync(new URL(`../fixtures/argent-0.29/${name}`, import.meta.url), "utf8"));
|
||||
// euxy's rename sheet on iOS 26 under Argent 0.29, its field focused and the keyboard up: the head of brentvatne/euxy#172
|
||||
// (review run 39362642343). Its root holds three windows: the app's, the input view placeholder's, and the keyboard's.
|
||||
const RENAME_KEYBOARD_UP = fixture("fixture-ios-ui-tree-rename.json");
|
||||
// The same sheet a moment earlier, as the base was captured: the field focused and its cursor showing, no keyboard yet.
|
||||
const RENAME_KEYBOARD_DOWN = { ...RENAME_KEYBOARD_UP, keyboardVisible: false,
|
||||
roots: [{ ...RENAME_KEYBOARD_UP.roots[0], children: RENAME_KEYBOARD_UP.roots[0].children.slice(0, 1) }] };
|
||||
const ANDROID_RENAME = fixture("fixture-android-ui-tree-rename.json");
|
||||
|
||||
const nodesOf = tree => snapshotFromTree(tree, "ios", 1).nodes;
|
||||
const labelsOf = nodes => parseControllerOutput("agent-device", JSON.stringify({ success: true, data: { nodes } })).elements.map(element => element.label);
|
||||
const nameOf = node => node.label ?? node.identifier;
|
||||
const rect = (x, y, width, height) => ({ x, y, width, height });
|
||||
|
||||
test("the iOS keyboard Argent reports is its own windows: the input view placeholder's and the keyboard's, Emoji and Dictate included", () => {
|
||||
const nodes = nodesOf(RENAME_KEYBOARD_UP);
|
||||
assert.deepEqual(keyboardBounds(nodes), rect(0, 539, 402, 335), "the input view and prediction bar down to the bottom edge");
|
||||
const owned = [...keyboardNodes(nodes)];
|
||||
const names = owned.map(nameOf).filter(Boolean);
|
||||
for (const name of ["inputView", "SystemInputAssistantView", "Typing Predictions", "Q", "shift", "Emoji", "Dictate"]) assert.ok(names.includes(name), name);
|
||||
assert.ok(owned.every(node => node.systemKeyboard === true), "only nodes of the keyboard's windows");
|
||||
// Every element of the app's window stays, those behind the keyboard's band among them.
|
||||
const app = nodesOf(RENAME_KEYBOARD_DOWN).map(nameOf).filter(Boolean);
|
||||
for (const name of app) assert.ok(nodes.some(node => nameOf(node) === name && !keyboardNodes(nodes).has(node)), name);
|
||||
assert.ok(app.includes("AdditionalDimmingOverlay") && app.includes("Tab Bar") && app.includes("Clear name"));
|
||||
});
|
||||
|
||||
test("a keyboard up on one side only leaves the elements and the accessibility fingerprint unchanged", () => {
|
||||
// In euxy#172 the review led with "Added button “Dictate”", “Emoji” and “inputView” over the label change.
|
||||
const up = nodesOf(RENAME_KEYBOARD_UP), down = nodesOf(RENAME_KEYBOARD_DOWN);
|
||||
assert.deepEqual(labelsOf(up), labelsOf(down));
|
||||
for (const name of ["Dictate", "Emoji", "inputView", "Q"]) assert.equal(labelsOf(up).includes(name), false, name);
|
||||
// The bridge numbers nodes in order, so the app's own nodes keep their indices when the keyboard's windows follow them.
|
||||
const snapshot = nodes => JSON.stringify({ success: true, data: { nodes: nodes.map(({ ref: _ref, ...node }) => node) } });
|
||||
assert.equal(normalizeSnapshot(snapshot(up)), normalizeSnapshot(snapshot(down)));
|
||||
});
|
||||
|
||||
test("the text field with the focus is passed on from Argent", () => {
|
||||
const up = nodesOf(RENAME_KEYBOARD_UP), down = nodesOf(RENAME_KEYBOARD_DOWN);
|
||||
assert.equal(up.filter(node => node.focused === true).map(node => node.type).join(), "TextField");
|
||||
assert.equal(focusedTextInput(up), true);
|
||||
assert.equal(focusedTextInput(down), true);
|
||||
assert.equal(keyboardBounds(down), undefined);
|
||||
assert.equal(focusedTextInput(nodesOf(fixture("fixture-ios-ui-tree-home.json"))), false);
|
||||
});
|
||||
|
||||
test("an element the app identifies as inputView in its own window is the app's", () => {
|
||||
// A chat composer with testID="inputView", no keyboard up.
|
||||
const app = RENAME_KEYBOARD_DOWN.roots[0].children[0];
|
||||
const composer = { role: "AXGroup", type: "Other", identifier: "inputView", frame: rect(0, 0.8, 1, 0.1),
|
||||
children: [{ role: "AXButton", type: "Button", label: "Send", frame: rect(0.8, 0.82, 0.15, 0.05), children: [] }] };
|
||||
const tree = { ...RENAME_KEYBOARD_DOWN, roots: [{ ...RENAME_KEYBOARD_DOWN.roots[0], children: [{ ...app, children: [...app.children, composer] }] }] };
|
||||
const nodes = nodesOf(tree);
|
||||
assert.equal(keyboardNodes(nodes).size, 0);
|
||||
assert.equal(keyboardBounds(nodes), undefined);
|
||||
assert.ok(labelsOf(nodes).includes("inputView") && labelsOf(nodes).includes("Send"));
|
||||
});
|
||||
|
||||
test("a second window of the app's own, its testID inputView included, is the app's; the keyboard's windows need a Keyboard", () => {
|
||||
// The app's second UIWindow (a toast or a floating composer) with a view identified as inputView and a Save button.
|
||||
const window = { role: "AXGroup", type: "Application", frame: rect(0, 0, 1, 1), children: [
|
||||
{ role: "AXGroup", type: "Other", identifier: "inputView", frame: rect(0, 0.8, 1, 0.1),
|
||||
children: [{ role: "AXButton", type: "Button", label: "Save", frame: rect(0.8, 0.82, 0.15, 0.05), children: [] }] }] };
|
||||
const [app, placeholder, keyboard] = RENAME_KEYBOARD_UP.roots[0].children;
|
||||
const withWindows = windows => nodesOf({ ...RENAME_KEYBOARD_UP, roots: [{ ...RENAME_KEYBOARD_UP.roots[0], children: [app, ...windows] }] });
|
||||
for (const windows of [[window], [window, keyboard], [window, placeholder, keyboard]]) {
|
||||
const labels = labelsOf(withWindows(windows));
|
||||
assert.ok(labels.includes("Save") && labels.includes("inputView"), `kept beside ${windows.length - 1} keyboard windows`);
|
||||
}
|
||||
// Without the keyboard's own window, an input view placeholder window proves nothing.
|
||||
assert.ok(labelsOf(withWindows([placeholder])).includes("inputView"));
|
||||
assert.equal(keyboardNodes(withWindows([placeholder])).size, 0);
|
||||
// With it, the placeholder is the keyboard's too, as in the real tree.
|
||||
assert.equal(labelsOf(withWindows([placeholder, keyboard])).includes("inputView"), false);
|
||||
});
|
||||
|
||||
test("agent-device's keyboard is its Keyboard element and what is under it, and its host bridge's keyboard view classes", () => {
|
||||
// XCTest: the Keyboard directly under the Application, beside the app's own controls in the same band.
|
||||
const xctest = [
|
||||
{ index: 0, type: "Application", label: "Example", rect: rect(0, 0, 402, 874) },
|
||||
{ index: 1, parentIndex: 0, type: "TextField", label: "Pattern name", focused: true, rect: rect(23, 386, 355, 50) },
|
||||
{ index: 2, parentIndex: 0, type: "Keyboard", rect: rect(0, 583, 402, 233) },
|
||||
{ index: 3, parentIndex: 2, type: "Key", label: "Q", rect: rect(5, 590, 39, 54) },
|
||||
{ index: 4, parentIndex: 0, type: "Button", label: "Home tab", rect: rect(0, 800, 134, 74) },
|
||||
{ index: 5, parentIndex: 0, type: "Other", label: "Formatting toolbar", rect: rect(0, 540, 402, 43) },
|
||||
{ index: 6, parentIndex: 0, type: "Button", label: "Emoji", rect: rect(8, 300, 68, 40) },
|
||||
];
|
||||
assert.deepEqual(keyboardBounds(xctest), rect(0, 583, 402, 233));
|
||||
assert.deepEqual(labelsOf(xctest), ["Example", "Pattern name", "Home tab", "Formatting toolbar", "Emoji"]);
|
||||
// The host bridge names the dock's buttons by class, wherever they sit.
|
||||
const hostBridge = [
|
||||
{ type: "TextField", role: "RCTUITextField", label: "Pattern name", focused: true, rect: rect(23, 386, 355, 50) },
|
||||
{ type: "Button", role: "UIKeyboardDockItemButton", label: "Dictate", rect: rect(325, 806, 68, 68) },
|
||||
{ type: "Other", role: "UIAccessibilityElementKBKey", label: "Q", rect: rect(5, 590, 39, 54) },
|
||||
{ type: "Button", role: "RCTViewComponentView", label: "Profile tab", rect: rect(268, 800, 134, 74) },
|
||||
];
|
||||
assert.ok(keyboardBounds(hostBridge));
|
||||
assert.deepEqual(labelsOf(hostBridge), ["Pattern name", "Profile tab"]);
|
||||
});
|
||||
|
||||
test("nothing without a parent link is taken for the keyboard's, and keys alone are no keyboard", () => {
|
||||
// agent-device's flat shape: no parentIndex. The Keyboard element goes; nothing can be proven to be under it.
|
||||
const flat = [
|
||||
{ type: "Keyboard", rect: rect(0, 583, 402, 233) },
|
||||
{ type: "Key", label: "Q", rect: rect(5, 590, 39, 54) },
|
||||
{ type: "Button", label: "Profile tab", rect: rect(268, 800, 134, 74) },
|
||||
];
|
||||
assert.deepEqual(labelsOf(flat), ["Q", "Profile tab"]);
|
||||
// An app's own PIN pad (accessibilityRole="keyboardkey") and its footer action, with no system keyboard up.
|
||||
const pinPad = [
|
||||
{ index: 0, type: "Application", label: "Example", rect: rect(0, 0, 402, 874) },
|
||||
{ index: 1, parentIndex: 0, type: "SecureTextField", label: "PIN", focused: true, rect: rect(23, 200, 355, 50) },
|
||||
...["1", "2", "3"].map((digit, offset) => ({ index: 2 + offset, parentIndex: 0, type: "Key", label: digit, rect: rect(offset * 134, 600, 134, 70) })),
|
||||
{ index: 5, parentIndex: 0, type: "Button", label: "App footer action", rect: rect(0, 800, 402, 74) },
|
||||
];
|
||||
assert.equal(keyboardNodes(pinPad).size, 0);
|
||||
assert.equal(keyboardBounds(pinPad), undefined);
|
||||
assert.deepEqual(labelsOf(pinPad), ["Example", "PIN", "1", "2", "3", "App footer action"]);
|
||||
});
|
||||
|
||||
test("in landscape the keyboard is told apart the same way, the app's bottom bar kept", () => {
|
||||
const landscape = [
|
||||
{ index: 0, type: "Application", label: "Example", rect: rect(0, 0, 874, 402) },
|
||||
{ index: 1, parentIndex: 0, type: "TextField", label: "Search", focused: true, rect: rect(100, 60, 674, 40) },
|
||||
{ index: 2, parentIndex: 0, type: "Keyboard", rect: rect(0, 230, 874, 172) },
|
||||
{ index: 3, parentIndex: 2, type: "Key", label: "Q", rect: rect(60, 236, 70, 38) },
|
||||
{ index: 4, parentIndex: 0, type: "TabBar", label: "Tab Bar", rect: rect(0, 352, 874, 50) },
|
||||
];
|
||||
assert.deepEqual(keyboardBounds(landscape), rect(0, 230, 874, 172));
|
||||
assert.deepEqual(labelsOf(landscape), ["Example", "Search", "Tab Bar"]);
|
||||
});
|
||||
|
||||
test("an Android input method's window is the keyboard under Argent, AOSP's included, and Argent says when one shows", () => {
|
||||
const ime = pkg => ({ role: "FrameLayout", type: "android.widget.FrameLayout", bundleId: pkg, frame: rect(0, 0.6, 1, 0.4),
|
||||
children: [{ role: "Button", type: "android.widget.Button", contentDescription: "q", frame: rect(0.01, 0.65, 0.09, 0.06), children: [] }] });
|
||||
const before = snapshotFromTree(ANDROID_RENAME, "android", 1, "dev.brent.euxy").nodes;
|
||||
assert.equal(focusedTextInput(before), true, "Argent marks the focused EditText");
|
||||
assert.equal(keyboardBounds(before), undefined);
|
||||
for (const pkg of ["com.google.android.inputmethod.latin", "com.android.inputmethod.latin"]) {
|
||||
const after = snapshotFromTree({ ...ANDROID_RENAME, keyboardVisible: true, roots: [...ANDROID_RENAME.roots, ime(pkg)] }, "android", 1, "dev.brent.euxy").nodes;
|
||||
assert.deepEqual(keyboardBounds(after), rect(0, 960, 720, 640), pkg);
|
||||
assert.deepEqual(labelsOf(after), labelsOf(before), pkg);
|
||||
}
|
||||
});
|
||||
|
||||
test("the wait reads until the keyboard is in one place twice, at least twice, at most six times", () => {
|
||||
const field = { type: "TextField", label: "Pattern name", focused: true, rect: rect(23, 386, 355, 50) };
|
||||
const keyboardAt = y => [field, { index: 9, type: "Keyboard", rect: rect(0, y, 402, 874 - y) }];
|
||||
// Already up when the screen was read: one more read must find it in the same place.
|
||||
const up = keyboardSettle({ nodes: keyboardAt(583) });
|
||||
assert.equal(up.more(0), true, "a keyboard already up may still be rising");
|
||||
up.read({ nodes: keyboardAt(583) });
|
||||
assert.deepEqual([up.outcome, up.reads, up.more(250)], ["shown", 1, false]);
|
||||
// Caught rising: it moves between reads, then holds.
|
||||
const rising = keyboardSettle({ nodes: [field] });
|
||||
for (const y of [700, 583, 583]) rising.read({ nodes: keyboardAt(y) });
|
||||
assert.deepEqual([rising.outcome, rising.reads], ["shown", 3]);
|
||||
// A slow controller: the second read is owed even after a second has passed.
|
||||
const slow = keyboardSettle({ nodes: [field] });
|
||||
slow.read({ nodes: [field] });
|
||||
assert.equal(slow.more(2_500), true);
|
||||
slow.read({ nodes: [field] });
|
||||
assert.deepEqual([slow.outcome, slow.more(5_000)], ["absent", false], "no keyboard came: captured without one");
|
||||
// A slow first read without a keyboard, then the keyboard first seen still moving: read on until it holds.
|
||||
const late = keyboardSettle({ nodes: [field] });
|
||||
late.read({ nodes: [field] });
|
||||
late.read({ nodes: keyboardAt(700) });
|
||||
assert.deepEqual([late.outcome, late.more(2_500)], ["moving", true], "a moving keyboard is not given up on after a second");
|
||||
late.read({ nodes: keyboardAt(583) });
|
||||
late.read({ nodes: keyboardAt(583) });
|
||||
assert.deepEqual([late.outcome, late.reads], ["shown", 4]);
|
||||
// But not past four seconds, however few reads that was.
|
||||
const stalled = keyboardSettle({ nodes: [field] });
|
||||
stalled.read({ nodes: [field] });
|
||||
stalled.read({ nodes: keyboardAt(700) });
|
||||
assert.equal(stalled.more(4_000), false);
|
||||
// Six reads at most, however fast they come.
|
||||
const busy = keyboardSettle({ nodes: [field] });
|
||||
for (let read = 0; busy.more(0); read++) busy.read({ nodes: keyboardAt(read % 2 ? 583 : 700) });
|
||||
assert.deepEqual([busy.outcome, busy.reads], ["moving", 6]);
|
||||
// The field let go and no keyboard shows: nothing more to wait for.
|
||||
const released = keyboardSettle({ nodes: [field] });
|
||||
released.read({ nodes: [{ ...field, focused: false }] });
|
||||
assert.deepEqual([released.outcome, released.more(0)], ["unfocused", false]);
|
||||
// Argent on Android says a keyboard shows without placing it: it shows twice running.
|
||||
const android = keyboardSettle({ nodes: [field] });
|
||||
android.read({ nodes: [field], keyboardVisible: true });
|
||||
android.read({ nodes: [field], keyboardVisible: true });
|
||||
assert.equal(android.outcome, "shown");
|
||||
assert.equal(keyboardSettle({ nodes: [{ ...field, focused: false }] }), undefined, "a screen without a focused field is not waited on");
|
||||
});
|
||||
@@ -679,6 +679,80 @@ test('a config without targets writes the same records and comment as before tar
|
||||
assert.deepEqual(Object.keys(readCommentRecords(after).current), ['ios', 'android']);
|
||||
});
|
||||
|
||||
/**
|
||||
* brentvatne/euxy#172, review run 39362642343: five targets, two strings changed. Tempo's footnote sits low in a
|
||||
* scrolling sheet. On the closed iPhone Duo the accessibility tree has the new text but the screenshots match: the
|
||||
* footnote runs past the bottom of the screen (`offScreen`, from the elements' bounds). On the half-open and opened Duo
|
||||
* neither the tree nor the pixels have it. Android captures no dark appearance.
|
||||
*/
|
||||
const euxyTargets = [{ id: 'iphone', platform: 'ios', label: 'iPhone 17' }, { id: 'duo-closed', platform: 'ios', label: 'iPhone Duo (closed)' },
|
||||
{ id: 'duo-half-open', platform: 'ios', label: 'iPhone Duo (half-open)' }, { id: 'duo-open', platform: 'ios', label: 'iPhone Duo (open)' },
|
||||
{ id: 'android', platform: 'android', label: 'Android' }];
|
||||
const footnote = 'Added text “Changes apply live while playing. Base resolution sets the step grid for lanes you add next.”.';
|
||||
const footnoteText = 'Added text “Changes apply live while playing\\. Base resolution sets the step grid for lanes you add next\\.”';
|
||||
function euxyReview(target, options = {}) {
|
||||
const own = { sourceImpact: ['own'], sourceScope: 'screen' };
|
||||
const matched = { verdict: 'unchanged', reason: 'matched', mismatchPercentage: 0, threshold: 0.1, tool: '[email protected]' };
|
||||
const changed = (description, mismatchPercentage) => ({ ...own,
|
||||
review: { status: 'modified', description, facts: [{ category: 'element', templateId: 'element.added', bullet: description }] },
|
||||
visual: { verdict: 'modified', reason: 'repeated-cross-build-delta', mismatchPercentage, threshold: 0.1, tool: '[email protected]' } });
|
||||
const inTreeOnly = { ...changed(footnote, 0), visual: matched };
|
||||
if (options.offScreen !== false) inTreeOnly.review.offScreen = true;
|
||||
if (options.diffFailed) inTreeOnly.visual = { ...matched, reason: 'diff-failed' };
|
||||
const unseen = { ...own, review: { status: 'unchanged', description: 'No visible change' }, visual: matched };
|
||||
const tempo = target === 'duo-closed' ? inTreeOnly : ['duo-half-open', 'duo-open'].includes(target) ? unseen : changed(footnote, 0.11);
|
||||
const rename = changed('Added text “Pattern name”.', 0.16);
|
||||
const schemes = target === 'android' ? [''] : ['', '@dark'];
|
||||
return review(schemes.flatMap(scheme => [
|
||||
surface(`tempo${scheme}`, 'modified', { route: '/tempo', extra: structuredClone(tempo) }),
|
||||
surface(`rename-pattern${scheme}`, 'modified', { route: '/rename-pattern', extra: structuredClone(rename) }),
|
||||
]));
|
||||
}
|
||||
async function euxyRows(reviewOf = euxyReview) {
|
||||
const { platformSectionMerge } = await import('../../dist/report/pr-comment-body.js');
|
||||
let body;
|
||||
for (const target of euxyTargets) body = platformSectionMerge(state({ artifactOnly: true, target, targets: euxyTargets, review: reviewOf(target.id),
|
||||
build: { ...state().build, platform: target.platform }, runtime: { ...state().runtime, platform: target.platform, deviceModel: target.label } }), target.id).merge(body);
|
||||
return { body, rows: listOf(body).map(line => line.replace(/\(<[^>]+>\)/, '')) };
|
||||
}
|
||||
|
||||
test('a change whose elements run past the screen on a device is named off-screen there, beside the devices that show it (euxy#172)', async () => {
|
||||
const { body, rows } = await euxyRows();
|
||||
assert.deepEqual(rows, [
|
||||
`- [\`/tempo\`] · ${footnoteText} · iPhone 17, Android only · off-screen on iPhone Duo \\(closed\\)`,
|
||||
`- [\`/rename-pattern\`] · Added text “Pattern name”`,
|
||||
`- [\`/tempo\`] (dark) · ${footnoteText} · iPhone 17 only · off-screen on iPhone Duo \\(closed\\)`,
|
||||
`- [\`/rename-pattern\`] (dark) · Added text “Pattern name” · not on Android`,
|
||||
], 'the half-open and opened Duo have no trace of it, so they stay among the devices it is not on');
|
||||
const { readCommentRecords } = await import('../../dist/report/pr-comment-body.js');
|
||||
assert.deepEqual(readCommentRecords(body).current['duo-closed'].changes.filter(change => change.offScreen).map(change => change.id), ['tempo', 'tempo@dark'],
|
||||
'the mark survives the comment round trip');
|
||||
});
|
||||
|
||||
test('without geometry that puts it past the screen, or with a failed comparison, a device keeps today\'s wording', async () => {
|
||||
// The review euxy#172 published: the closed Duo's change is accessibility-only, with nothing saying where its elements are.
|
||||
const { rows } = await euxyRows(target => euxyReview(target, { offScreen: false }));
|
||||
assert.equal(rows[0], `- [\`/tempo\`] · ${footnoteText} · not on iPhone Duo \\(half\\-open\\), iPhone Duo \\(open\\)`);
|
||||
assert.equal(rows[2], `- [\`/tempo\`] (dark) · ${footnoteText} · iPhone 17, iPhone Duo \\(closed\\) only`);
|
||||
// A comparison that failed is no verdict at all: the closed Duo is not counted among the devices with the change.
|
||||
const failed = await euxyRows(target => euxyReview(target, { diffFailed: true }));
|
||||
assert.match(failed.rows[0], / · on iPhone 17, Android$/);
|
||||
assert.doesNotMatch(failed.rows.join('\n'), /off-screen/);
|
||||
});
|
||||
|
||||
test('a change one platform has and another lacks keeps "only" and "not on"', async () => {
|
||||
// An iOS-only branch in a shared screen file: Android's capture of the screen shows no change, and says nothing off-screen.
|
||||
const iosOnly = target => {
|
||||
const reviewed = euxyReview(target);
|
||||
if (target === 'android') for (const entry of reviewed.surfaces) Object.assign(entry, { review: { status: 'unchanged', description: 'No visible change' },
|
||||
visual: { verdict: 'unchanged', reason: 'matched', mismatchPercentage: 0, threshold: 0.1, tool: '[email protected]' } });
|
||||
return reviewed;
|
||||
};
|
||||
const { rows } = await euxyRows(iosOnly);
|
||||
assert.equal(rows[0], `- [\`/tempo\`] · ${footnoteText} · iPhone 17 only · off-screen on iPhone Duo \\(closed\\)`);
|
||||
assert.match(rows.find(row => row.startsWith('- [`/rename-pattern`] ·')), / · not on Android$/);
|
||||
});
|
||||
|
||||
test('several targets share a comment: named in config order in the subtitle, each linking its report, keyed by id in the records and history', async () => {
|
||||
const { platformSectionMerge, readCommentRecords } = await import('../../dist/report/pr-comment-body.js');
|
||||
const [iphone, closed, open, android] = duoTargets;
|
||||
|
||||
@@ -320,3 +320,15 @@ test("an excluded surface carries no capture evidence", () => {
|
||||
excluded.surfaces[1].excluded = false;
|
||||
assert.throws(() => assertPublicReview(excluded), /excluded surface carries capture evidence/);
|
||||
});
|
||||
|
||||
test("a review says its change is off-screen only on a modified verdict whose screenshots match", () => {
|
||||
const offScreen = (status, verdict) => {
|
||||
const input = review();
|
||||
input.surfaces[0].review = { status, description: status === "modified" ? "Added text “lanes you add next”." : "No visible change", offScreen: true };
|
||||
input.surfaces[0].visual = { verdict, reason: verdict === "unchanged" ? "matched" : "repeated-cross-build-delta", mismatchPercentage: verdict === "unchanged" ? 0 : 3, threshold: 0.1, tool: "[email protected]" };
|
||||
return input;
|
||||
};
|
||||
assert.doesNotThrow(() => assertPublicReview(offScreen("modified", "unchanged")));
|
||||
assert.throws(() => assertPublicReview(offScreen("modified", "modified")), /inconsistent/);
|
||||
assert.throws(() => assertPublicReview(offScreen("unchanged", "unchanged")), /inconsistent/);
|
||||
});
|
||||
@@ -14,13 +14,62 @@ function easFake(listed) {
|
||||
return { calls, eas };
|
||||
}
|
||||
|
||||
test("a session's control actions come from --list, by session id; an older session or CLI has none", () => {
|
||||
test("a session's control actions come from --list, by session id", () => {
|
||||
const { calls, eas } = easFake(JSON.stringify({ "appearance.set": {}, "springboard.restart": {} }));
|
||||
assert.deepEqual([...simulatorControlActions(eas, "session-1")], ["appearance.set", "springboard.restart"]);
|
||||
assert.deepEqual(calls[0].args, ["simulator:control", "--list", "--id", "session-1", "--json", "--non-interactive"]);
|
||||
assert.equal(calls[0].options.allowFailure, true);
|
||||
assert.equal(simulatorControlActions(easFake("Unknown command simulator:control").eas, "session-1").size, 0);
|
||||
assert.equal(simulatorControlActions(easFake(new Error("eas exited 1")).eas, "session-1").size, 0);
|
||||
});
|
||||
|
||||
test("a session that lists no actions, or whose serve-sim or EAS CLI predates the list, has none, and is asked once", () => {
|
||||
const pauses = [];
|
||||
const options = { sleep: milliseconds => pauses.push(milliseconds) };
|
||||
const empty = easFake("{}");
|
||||
assert.equal(simulatorControlActions(empty.eas, "session-1", options).size, 0);
|
||||
assert.equal(empty.calls.length, 1);
|
||||
// eas-cli 25.2.0 on a session whose serve-sim refuses `actions.describe`, and an EAS CLI without simulator:control.
|
||||
for (const reason of ["This simulator session runs a serve-sim version that does not support this action. Start a new session with `eas simulator:start` to get a newer version.",
|
||||
"The simulator did not return its list of actions. Start a new session with `eas simulator:start` to get a newer serve-sim version."]) {
|
||||
const calls = [];
|
||||
const eas = args => { calls.push(args); return { stdout: "", stderr: `${reason}\nError: simulator:control command failed.\n`, status: 1 }; };
|
||||
assert.equal(simulatorControlActions(eas, "session-1", options).size, 0);
|
||||
assert.equal(calls.length, 1);
|
||||
}
|
||||
assert.equal(simulatorControlActions(easFake(new Error("eas exited with status 2: command simulator:control not found")).eas, "session-1", options).size, 0);
|
||||
assert.deepEqual(pauses, []);
|
||||
});
|
||||
|
||||
// One failed list made the dark session of euxy's main baseline run 39487618529 look as if it offered no actions.
|
||||
function flakyList(failures, failure = () => ({ stdout: "", stderr: "The simulator preview server closed the connection before it answered.\nError: simulator:control command failed.\n", status: 1 })) {
|
||||
const calls = [];
|
||||
const eas = args => {
|
||||
calls.push(args);
|
||||
if (calls.length <= failures) return failure(calls.length);
|
||||
return { stdout: JSON.stringify({ "appearance.set": {}, "app.approveUrlScheme": {}, "springboard.restart": {} }), stderr: "", status: 0 };
|
||||
};
|
||||
return { calls, eas };
|
||||
}
|
||||
|
||||
test("a list that fails once is asked for again after a pause, and its actions are kept", () => {
|
||||
for (const failure of [undefined, () => { throw new Error("eas simulator:control exited with status 1"); }, () => ({ stdout: "Connecting", stderr: "" })]) {
|
||||
const { calls, eas } = flakyList(1, failure);
|
||||
const pauses = [];
|
||||
assert.deepEqual([...simulatorControlActions(eas, "session-1", { sleep: milliseconds => pauses.push(milliseconds) })], ["appearance.set", "app.approveUrlScheme", "springboard.restart"]);
|
||||
assert.equal(calls.length, 2);
|
||||
assert.deepEqual(calls[1], calls[0]);
|
||||
assert.equal(pauses.length, 1);
|
||||
}
|
||||
});
|
||||
|
||||
test("a list that keeps failing throws a retryable error naming the session and EAS's reason, never an empty set", () => {
|
||||
const { calls, eas } = flakyList(Infinity);
|
||||
const pauses = [];
|
||||
assert.throws(() => simulatorControlActions(eas, "session-1", { sleep: milliseconds => pauses.push(milliseconds) }), error =>
|
||||
error.code === "SIMULATOR_CONTROL_UNAVAILABLE" && error.retryable === true && error.stage === "session"
|
||||
&& /session-1/.test(error.message) && /closed the connection before it answered/.test(error.message) && !/command failed/.test(error.message));
|
||||
assert.equal(calls.length, 3);
|
||||
assert.equal(pauses.length, calls.length - 1);
|
||||
assert.ok(pauses.every(milliseconds => milliseconds > 0 && milliseconds <= 10_000));
|
||||
});
|
||||
|
||||
test("an action's values follow --, so one that starts with a dash is never read as a flag", () => {
|
||||
|
||||
@@ -97,3 +97,16 @@ test("accessibility fingerprints ignore Android system UI nodes", async () => {
|
||||
assert.equal(sameApp.accessibility, base.accessibility);
|
||||
assert.notEqual(changedApp.accessibility, base.accessibility);
|
||||
});
|
||||
|
||||
test("accessibility fingerprints are versioned, and leave out the system keyboard and which field has the focus", async () => {
|
||||
const snapshot = (focused, keyboard) => JSON.stringify({ success: true, data: { nodes: [
|
||||
{ index: 0, type: "Application", label: "euxy", rect: { x: 0, y: 0, width: 402, height: 874 } },
|
||||
{ index: 1, parentIndex: 0, type: "TextField", label: "Pattern name", ...(focused ? { focused: true } : {}), rect: { x: 23, y: 386, width: 355, height: 50 } },
|
||||
...(keyboard ? [{ index: 2, parentIndex: 0, type: "Keyboard", rect: { x: 0, y: 583, width: 402, height: 233 } },
|
||||
{ index: 3, parentIndex: 2, type: "Key", label: "Q", rect: { x: 5, y: 590, width: 39, height: 54 } }] : []),
|
||||
] } });
|
||||
const resting = await fingerprintState({ snapshotText: snapshot(false, false) });
|
||||
assert.match(resting.accessibility, /^a2:[0-9a-f]{64}$/);
|
||||
assert.equal((await fingerprintState({ snapshotText: snapshot(true, false) })).accessibility, resting.accessibility, "focus alone does not change the tree");
|
||||
assert.equal((await fingerprintState({ snapshotText: snapshot(true, true) })).accessibility, resting.accessibility, "nor does the keyboard");
|
||||
});
|
||||
Reference in new issue
Block a user