Compare commits

...
4 Commits
25 changed files with 952 additions and 73 deletions

No files matched your search

+16 -5
View File
@@ -14,7 +14,7 @@ Source of truth: `templates/explore-review.yml`, `templates/explore-baseline.yml
| Pipeline | `capture:review` → `runCiReview` | The same `runCiReview`, called directly |
| EAS credentials | `EXPO_TOKEN` secret | Local `eas login` session or `EXPO_TOKEN` |
| Model credentials (agent mode) | `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` | Claude Code or Codex subscription login; API keys are removed from the subprocess |
| Base cache | A baseline workflow artifact for the exact base SHA, when one exists | None; both sides are captured |
| Base cache | A baseline workflow artifact for the exact base SHA, when one exists, or the baseline run still capturing it, waited for alongside head | None; both sides are captured |
| Output | Report artifact + sticky PR comment with **Open** / **zip** | `runs/<id>/artifacts/report.zip` + `PR-comment.md`, opened with `ci:local:view` |
## 1. Installing Explore in an app repository
@@ -49,8 +49,8 @@ Each workflow starts with a `targets` job: it installs the pinned CLI and runs `
| | checkout `app-base`, `app-head` | Checks out the exact base and head SHAs with full history and without persisted credentials. |
| | Verify deterministic source planning | `source:verify` builds the capture plan from source alone in two fresh processes and compares the bytes. Runs before any build or simulator work and needs no credentials. |
| | Install app dependencies | `npm ci --ignore-scripts` in both checkouts (`APP_INSTALL_DIR` for monorepos). |
| | Find / Download optional exact-base evidence | `ci:find-baseline` looks for a finished run's `explore-baseline-<base sha>-<target>` artifact (a run another target's baseline failed in counts: a baseline job uploads its artifact only after its own steps passed) (target `ios` also accepts the older unsuffixed `explore-baseline-<base sha>`) from the baseline workflow on the default branch and downloads it as `EXPLORE_CAPTURE_CACHE`. Missing evidence is not an error. |
| | Capture review | `capture:review` (§3). `EXPO_TOKEN`, the provider key and the app's `{{secret:NAME}}` values reach only this step. The sticky PR comment is updated while it runs. It writes `preview=<hosting>` to the step outputs, which decides whether the deploy job runs. |
| | Find / Download optional exact-base evidence | `ci:find-baseline` looks for a finished run's `explore-baseline-<base sha>-<target>` artifact (a run another target's baseline failed in counts: a baseline job uploads its artifact only after its own steps passed) (target `ios` also accepts the older unsuffixed `explore-baseline-<base sha>`) from the baseline workflow on the default branch and downloads it as `EXPLORE_CAPTURE_CACHE`. With none, it names a baseline run of the base commit still queued or capturing (`pending`), held to the same checks: the baseline workflow, the default branch of this repository, a `push` or `workflow_dispatch`, exactly the base SHA; nothing from the PR head selects it. Missing evidence and a failed lookup are not errors. |
| | Capture review | `capture:review` (§3). With `EXPLORE_BASELINE_RUN` (the find step's `pending`) it waits for that run's baseline of this target (§3.1). `EXPO_TOKEN`, the provider key and the app's `{{secret:NAME}}` values reach only this step. The sticky PR comment is updated while it runs. It writes `preview=<hosting>` to the step outputs, which decides whether the deploy job runs. |
| | Stop capture sessions and check the checkouts | `ci:after-capture`, whenever the PR was resolved, success or not: stops the EAS Simulator sessions named in `head-session` / `base-session`, nothing else, then fails with `CHECKOUT_MODIFIED` if capture modified an app checkout. |
| | Upload captured review / diagnostics | `explore-capture-<run>-<attempt>-<target>` feeds the report job. On failure `explore-diagnostics-<run>-<attempt>-<target>` keeps plans, exports and agent diagnostics for 7 days. |
| report (matrix: target) | Download the capture, install Explore | Runs without Expo or model credentials, after every capture job, whether or not each succeeded (`!cancelled()`). Each leg downloads its own target's capture by an exact-name pattern: a target whose capture failed uploaded none, so its leg packages and publishes nothing, and the other targets' reports still reach the comment. |
@@ -68,7 +68,7 @@ Validate baseline source → Install Explore → Verify deterministic source pla
### 2.4 `explore-compare.yml`
The same jobs as a PR review, for any `base` and `head` without a pull request. The `targets` job resolves both refs once with `ci:compare-refs` (a tag wins over a branch of the same name; the same commit twice fails with `COMPARE_SAME_COMMIT`) and passes the comparison to the capture jobs, which run `capture:review --compare` with no GitHub requests. The optional `scope` input overrides `exploration.scope` for that run; `expires-in` is accepted and ignored. The report job's `ci:publish` writes each target's **Open** and **zip** links to the run summary, and the deploy job adds the hosted link there; nothing comments on a pull request.
The same jobs as a PR review, for any `base` and `head` without a pull request. The `targets` job resolves both refs once with `ci:compare-refs` (a tag wins over a branch of the same name; the same commit twice fails with `COMPARE_SAME_COMMIT`) and passes the comparison to the capture jobs, which run `capture:review --compare`; its only GitHub requests are the wait for a baseline run still capturing the base (§3.1), with the job's `actions: read` token. The optional `scope` input overrides `exploration.scope` for that run; `expires-in` is accepted and ignored. The report job's `ci:publish` writes each target's **Open** and **zip** links to the run summary, and the deploy job adds the hosted link there; nothing comments on a pull request.
## 3. Inside `capture:review` (`runCiReview`)
@@ -77,7 +77,7 @@ The same jobs as a PR review, for any `base` and `head` without a pull request.
| 1 | `plan` | `buildVerifiedPlan`: snapshot both sources, compute the two-dot diff, run `plan-worker` twice with reversed input order and compare the canonical plans byte for byte. Summarises `sourceChange` per surface into `execution-plan.json`; the PR comment shows the planned capture count and warnings. | `SOURCE_ANALYSIS_FAILED`, inconsistent plan |
| 2 | `preflight` | Checks `EXPO_TOKEN`. `prepareCiExploration`: nothing for deterministic mode; for agent mode, confirms the instructions file is committed, creates the provider session and makes one short tool-free model request. | `AGENT_API_KEY_MISSING`, `AGENT_INSTRUCTIONS_UNCOMMITTED`, `AGENT_MODEL_NOT_FOUND` |
| 3 | `capture-head` | `runCiMap(head)` (§4). Records head's runtime profile and build. | `CAPTURE_*`, `SETUP_*`, `EAS_*` |
| 4 | `capture-base` | Tries the cache first: a baseline is reused only when its recipe matches (tools, config, instructions, device, profile, observer hash, runtime profile). Only routes missing from the cache run through `runCiMap(base)`. | same |
| 4 | `capture-base` | Tries the cache first: a baseline is reused only when its recipe matches (tools, config, instructions, device, profile, observer hash, runtime profile). Only routes missing from the cache run through `runCiMap(base)`. With no cache but a baseline run still capturing the base, base starts only once the wait for it ends (§3.1). | same |
| 5 | `compare` | `assessPublicEvidence` compares pixels and accessibility for route pairs verified on both sides; `prepareCaptureReview` assigns verdicts (§6). | |
| 6 | `capture-validation` | Every selected route was captured on both sides. | `CAPTURE_INCOMPLETE` |
| 7 | `comparison-validation` | Every pair was compared visually and for accessibility. | `COMPARISON_INCOMPLETE` |
@@ -86,6 +86,17 @@ The same jobs as a PR review, for any `base` and `head` without a pull request.
A failure at any stage is recorded in `state.failure`, a partial report is written when a plan exists, and the PR comment switches to the failed state.
### 3.1 Waiting for a baseline still being captured
Right after a merge, the next pull request's review starts while the baseline workflow is still capturing the new base: in brentvatne/euxy's review run 39348956443 it had started 15 s before, and every target captured its base too (10–14 minutes per target instead of about 6, on twice the EAS Simulator sessions). A baseline job takes 6–8 minutes and head capture 5–8, so `capture:review` waits for the baseline beside head instead of before it (`src/ci/baseline-wait.ts`):
1. The wait starts after preflight, only when `EXPLORE_BASELINE_RUN` names a run, no downloaded baseline matches, base has routes to capture, and `capture.baselineWaitMinutes` (default 15, 0 never waits) is not 0. Head starts capturing at once.
2. Every 20 s it reads the run (checked again by the same trust rules as `ci:find-baseline`), this target's artifact (`explore-baseline-<sha>-<target>`) in that run, and the run's jobs, logging `waiting for main's baseline run <id> of <sha> (<target>: in progress, 3 min)` once a minute or when the state changes. Each read retries GitHub's 5xx and network failures (`retryGitHubLookup`).
3. When the artifact is listed, it is downloaded in-process (the token goes to GitHub, never to the storage URL it redirects to), unpacked into `<output>/baseline-download` within the cache's size limits, and restored as a downloaded baseline once head has its runtime profile. Another target's failed job does not matter; each target's job uploads its own artifact.
4. The wait ends without a baseline when this target's job fails, is cancelled or is skipped, the run ends without listing the artifact for a minute, the time runs out, GitHub refuses the token (`actions: read` missing) or keeps failing, or head fails. Base then captures at once, beside head when the sides capture concurrently. The wait never fails the review.
Head capture is never delayed. When the baseline arrives while head captures, base costs only its restore; when it does not, base starts the moment the wait gives up, so a failed baseline delays base only by the time it took to fail. Sessions are planned as without a wait: with a share of two or more, head keeps its half while base waits, so base can start in the rest; with one session, base follows head as before, and the wait may go on past head, up to its limit.
## 4. Capturing one side (`runCiMap`)
| # | Stage | Work |
+1 -1
View File
@@ -492,7 +492,7 @@ jobs:
- The run summary links each target's report (**Open** and **zip**, like a PR comment) and, with hosting, the hosted report.
- `setup --workflows` installs it from 0.19.0; `npx @expo/explore@latest setup --workflows` adds it to an existing setup. The caller grants `contents: read` and `actions: read`; `secrets: inherit` passes `EXPO_TOKEN` and any app-test secrets.
- Call it once per workflow run. A called workflow runs inside its caller's run, so a second call in the same run would reuse the first one's artifact names; use separate workflows instead.
- When the baseline workflow captured `base` in the last 30 days, with the same config and Explore version, that capture is reused and only `head` is captured. The report artifact is kept 30 days; a hosted link lasts as long as its EAS Hosting deployment (an aliased one until the alias moves).
- When the baseline workflow captured `base` in the last 30 days, with the same config and Explore version, that capture is reused and only `head` is captured. When it is still capturing `base` (right after a merge), the review waits for it while `head` captures, up to `capture.baselineWaitMinutes` (15 by default). The report artifact is kept 30 days; a hosted link lasts as long as its EAS Hosting deployment (an aliased one until the alias moves).
</details>
+1
View File
@@ -105,6 +105,7 @@ capture:
maxActions: 500
# maxSessions: 5 # the most EAS Simulator sessions one review runs at once, across all its targets and sides (default: 5, a typical plan's concurrent simulator limit). Each target's job gets floor(maxSessions / targets) sessions, at least one, and sessions left over go to foldable targets (pose): one target gets 5 (head and base at once, light and dark in their own sessions when that pays), two get 2 each, five get 1 each (head, then base). Below the number of targets, only that many target jobs run at once (the workflow's max-parallel, and a local review's targets), one session each. A side spreads its light and dark passes over separate sessions only when that saves it at least a minute, so the ceiling is a limit, not a target: a 3-route target stays on two sessions. euxy's four targets: 8 is two per target, 16 captures each appearance in its own session (measured: an iPhone side 6:12 → 4:26, one repack shared by its sessions, ~3.5 session-minutes more per side).
# On a starter plan (one simulator per platform), set maxSessions to the plan's limit, e.g. 2 for an iOS and an Android target.
# baselineWaitMinutes: 15 # a review that starts while the default branch's baseline run of its base commit is still capturing (right after a merge) waits up to this long for that target's baseline instead of capturing the base too (default 15; 0 never waits). Head captures meanwhile; base waits only where it would start, and captures itself if the baseline job fails or is cancelled, or the time runs out.
repack: eas # eas (hosted workflow job; the default on EAS Simulator) | local (repack-app on the runner; the default with simulator: local)
appearance: auto # auto (follow the app's userInterfaceStyle: automatic captures light and dark) | dark | light | [dark, light]
launch: screenshot # screenshot (relaunch the app per appearance and keep its splash and first settled frame, default) | video (also keep a launch recording in the report) | off
+2 -1
View File
@@ -111,6 +111,7 @@ capture:
maxActions: 500
# maxSessions: 5 # the most EAS Simulator sessions one review runs at once, across all its targets and sides (default: 5, a typical plan's concurrent simulator limit). Each target's job gets floor(maxSessions / targets) sessions, at least one, and sessions left over go to foldable targets (pose): one target gets 5 (head and base at once, light and dark in their own sessions when that pays), two get 2 each, five get 1 each (head, then base). Below the number of targets, only that many target jobs run at once (the workflow's max-parallel, and a local review's targets), one session each. A side spreads its light and dark passes over separate sessions only when that saves it at least a minute, so the ceiling is a limit, not a target: a 3-route target stays on two sessions. euxy's four targets: 8 is two per target, 16 captures each appearance in its own session (measured: an iPhone side 6:12 → 4:26, one repack shared by its sessions, ~3.5 session-minutes more per side).
# On a starter plan (one simulator per platform), set maxSessions to the plan's limit, e.g. 2 for an iOS and an Android target.
# baselineWaitMinutes: 15 # a review that starts while the default branch's baseline run of its base commit is still capturing (right after a merge) waits up to this long for that target's baseline instead of capturing the base too (default 15; 0 never waits). Head captures meanwhile; base waits only where it would start, and captures itself if the baseline job fails or is cancelled, or the time runs out.
repack: eas # eas (hosted EAS workflow job, the default on EAS Simulator) | local (repack-app on the runner, the default with simulator: local; --repack overrides it for one run)
appearance: auto # auto (follow the app's userInterfaceStyle: automatic captures light and dark) | dark | light | [dark, light]
launch: screenshot # screenshot (relaunch the app per appearance and keep its splash and first settled frame, default) | video (also keep a launch recording in the report) | off
@@ -391,7 +392,7 @@ The review workflow resolves the current PR through GitHub, checks out its exact
| Review packaging | The bundle contains metadata and assets, not a standalone HTML page |
| Publication | The artifact is uploaded before current-PR identity is checked again and the **Open** and **zip** links are posted |
The baseline workflow runs on default-branch pushes or manual default-branch dispatch. It captures one complete commit without needing a PR. Reports and baselines are retained for 30 days; intermediate evidence and diagnostics for 7 days.
The baseline workflow runs on default-branch pushes or manual default-branch dispatch. It captures one complete commit without needing a PR. A review that starts while the baseline of its base commit is still being captured, as the next pull request's review does right after a merge, waits for it (`capture.baselineWaitMinutes`, 15 minutes by default) while head captures, instead of capturing the base too. Reports and baselines are retained for 30 days; intermediate evidence and diagnostics for 7 days.
`exploration.scope: changes` includes source-affected routes and the routes one navigation step before or after them, so a reviewer sees how each change is reached and where it leads. The context is not transitive: in most apps tabs, guards and Home connect every screen, so all ancestors and descendants would be the whole app. A layout or screen guard that sends every signed-out screen to sign-in is not a step. Forwarding routes have no screenshot of their own, so their redirect chains are followed to the first screen. Unresolved navigation still selects every route. `all` captures the full inventory.
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@expo/explore",
"version": "0.20.3",
"version": "0.20.5",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@expo/explore",
"version": "0.20.3",
"version": "0.20.5",
"license": "MIT",
"dependencies": {
"@expo/repack-app": "0.10.1",
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@expo/explore",
"version": "0.20.3",
"version": "0.20.5",
"description": "Source-driven mobile app maps, deterministic CI capture, and visual reviews",
"license": "MIT",
"type": "module",
+24 -3
View File
@@ -360,9 +360,10 @@ async function executeCiMap(options: CiMapOptions, setStage: (stage: string) =>
let captureFailure: unknown;
const finish = async (): Promise<CiMapResult> => {
if (!(await store.exists())) await store.initialize(map, false);
// A session making only secondary passes owns the `<id>@<scheme>` frames of those passes, not the primary ones.
// A session making only secondary passes owns the `<id>@<scheme>` frames of those passes, not the primary ones, even
// when it could not switch to them and so captured none.
const ownFrameIds = options.appearancePass && !options.appearancePass.capture.includes(colorSchemes[0]!)
? selectedIds.flatMap(id => colorSchemes.filter(scheme => options.appearancePass!.capture.includes(scheme)).map(scheme => schemeScreenId(id, scheme)))
? selectedIds.flatMap(id => options.appearancePass!.capture.filter(scheme => scheme !== colorSchemes[0]).map(scheme => schemeScreenId(id, scheme)))
: selectedIds;
const incomplete = captureFailure !== undefined || ownFrameIds.some(id => outcomes[id]?.status !== "captured" || outcomes[id]?.landing !== "verified")
|| map.gaps.some(gap => gap.status === "open");
@@ -751,8 +752,28 @@ async function executeCiMap(options: CiMapOptions, setStage: (stage: string) =>
if (!handoff.connection) throw new Error("Simulator handoff is missing its controller connection");
connection = handoff.connection;
await rm(dotenvPath, { force: true });
if (options.platform === "ios") controlActions = simulatorControlActions(sessionEas, sessionId);
if (options.platform === "ios") controlActions = simulatorControlActions(sessionEas, sessionId, controlOptions);
if (!controlActions.has("appearance.set")) narrowToArgentScheme();
// A side split over sessions gives this one only some of its appearances (captureSideInSessions). One that cannot
// switch to any of them has nothing to capture, so it stops here, before the app is installed or the device folded:
// in brentvatne/euxy's main baseline run 39487618529, the dark session folded the device for frames it would never
// take, and its failed fold failed the target that the light session had captured.
const pass = options.appearancePass;
if (pass && !colorSchemes.some(scheme => pass.capture.includes(scheme))) {
const skipped = pass.capture.join(", ");
// The export merge needs the primary appearance's frames; a session that cannot take them fails the side.
if (pass.capture.includes(pass.all[0]!)) throw new CiError({ code: "SESSION_APPEARANCE_UNAVAILABLE", stage: "session", retryable: true,
message: `Simulator session ${sessionId} cannot switch to ${skipped}, the primary appearance it was given to capture.`,
nextAction: "Rerun the capture on a new session, or set capture.appearance to the appearance this controller renders." });
say(`session ${sessionId} cannot switch to ${skipped}, the appearance it was given; it stops without capturing, and the report has no ${skipped} frames`);
const error = { code: "SESSION_APPEARANCE_UNAVAILABLE", stage: "session", retryable: true,
message: `Simulator session ${sessionId} could not switch to ${skipped}.`, nextAction: "Rerun the capture on a new session." };
for (const id of selectedIds) for (const scheme of pass.capture) {
outcomes[schemeScreenId(id, scheme)] = { status: "failed", attempts: 0, landing: "unverified", reason: error.message, error };
}
// The finally block below stops the session after the export.
return await finish();
}
}
const controllerConfig = join(simulatorProject, "agent-device.json");
await writeFile(controllerConfig, "{}\n", { flag: "wx", mode: 0o400 });
+56 -17
View File
@@ -1,24 +1,11 @@
import { CiError } from "../shared/ci-errors.js";
import { isRecord } from "./common.js";
/** Runs the pinned EAS CLI with these arguments; throws when it exits non-zero unless `allowFailure` is set. */
export type EasCommand = (args: string[], options?: { timeoutMs?: number; allowFailure?: boolean }) => { stdout: string; stderr?: string };
/**
* The host actions an EAS Simulator iOS session offers through `eas simulator:control` (serve-sim's actions, 0.10.0 and
* later for the ones Explore uses): an empty set when the session or the EAS CLI offers none, so a caller falls back to
* what it did before them. The session is always named by id, never taken from a `.env.eas-simulator` beside it.
*
* These are a fallback: what the session's controller can do itself (agent-device or Argent: fold, appearance, a
* reader reset) goes through the controller. Explore uses an action here only for what neither controller offers —
* approving a link scheme, restarting SpringBoard, and Argent's appearance (software-mansion/argent#1509).
* Runs the pinned EAS CLI with these arguments; throws when it exits non-zero unless `allowFailure` is set. A runner that
* knows the exit status reports it as `status`.
*/
export function simulatorControlActions(eas: EasCommand, sessionId: string): Set<string> {
try {
const listed = eas(["simulator:control", "--list", "--id", sessionId, "--json", "--non-interactive"], { timeoutMs: 2 * 60_000, allowFailure: true });
const actions = JSON.parse(listed.stdout) as unknown;
return new Set(isRecord(actions) ? Object.keys(actions) : []);
} catch { return new Set(); }
}
export type EasCommand = (args: string[], options?: { timeoutMs?: number; allowFailure?: boolean }) => { stdout: string; stderr?: string; status?: number | null };
/**
* Pauses before each resend of a failed action. Every action Explore runs is idempotent, and the session's host can fail
@@ -27,6 +14,58 @@ export function simulatorControlActions(eas: EasCommand, sessionId: string): Set
*/
const SIMULATOR_CONTROL_RETRY_DELAYS_MS = [3_000, 6_000] as const;
/**
* What eas-cli 25.2.0 says when the session's serve-sim predates the action list (it refuses `actions.describe`, or
* answers it without a catalog), and what an EAS CLI without `simulator:control` says. Those sessions offer no actions,
* and asking again would not change that.
*/
const NO_ACTION_LIST = /serve-sim version that does not support this action|did not return its list of actions|command simulator:control not found/;
/**
* The host actions an EAS Simulator iOS session offers through `eas simulator:control` (serve-sim's actions, 0.10.0 and
* later for the ones Explore uses): an empty set when the session's serve-sim or the EAS CLI predates the list, so a
* caller falls back to what it did before them. The session is always named by id, never taken from a
* `.env.eas-simulator` beside it.
*
* These are a fallback: what the session's controller can do itself (agent-device or Argent: fold, appearance, a
* reader reset) goes through the controller. Explore uses an action here only for what neither controller offers —
* approving a link scheme, restarting SpringBoard, and Argent's appearance (software-mansion/argent#1509).
*
* eas-cli prints the list as JSON on stdout only when it succeeds; a failure prints its reason on stderr. A list that
* fails is asked for again after a pause, and one that still fails throws a retryable error rather than reading as "no
* actions": in brentvatne/euxy's main baseline run 39487618529, one failed list made the dark-appearance session look as
* if it offered none, so it dropped dark, approved no link scheme, and skipped the SpringBoard restart its fold needed,
* which failed the target. A session whose host cannot answer three times in a row would fail those actions too, and
* capturing on without them publishes a report that silently lacks an appearance; a new session is the way out.
*/
export function simulatorControlActions(eas: EasCommand, sessionId: string, options: SimulatorControlOptions = {}): Set<string> {
const sleep = options.sleep ?? (milliseconds => { Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, milliseconds); });
const args = ["simulator:control", "--list", "--id", sessionId, "--json", "--non-interactive"];
for (let retry = 0; ; retry += 1) {
let failure: string;
try {
const listed = eas(args, { timeoutMs: 2 * 60_000, allowFailure: true });
const actions = (() => { try { return JSON.parse(listed.stdout) as unknown; } catch { return undefined; } })();
if ((listed.status ?? 0) === 0 && isRecord(actions)) return new Set(Object.keys(actions));
const output = `${listed.stderr ?? ""}\n${listed.stdout}`;
if (NO_ACTION_LIST.test(output)) return new Set();
// eas-cli ends a failure with "simulator:control command failed."; the line before it says why.
failure = output.split("\n").map(line => line.trim()).filter(line => line && !/command failed\.?$/.test(line)).at(-1)?.slice(0, 300) || "it printed no list";
} catch (error) {
if (NO_ACTION_LIST.test(error instanceof Error ? error.message : String(error))) return new Set();
failure = firstLine(error);
}
const delayMs = SIMULATOR_CONTROL_RETRY_DELAYS_MS[retry];
if (delayMs === undefined) throw new CiError({ code: "SIMULATOR_CONTROL_UNAVAILABLE", stage: "session", retryable: true,
message: `EAS could not list the simulator:control actions of session ${sessionId} after ${retry + 1} attempts: ${failure}`,
nextAction: "Rerun the capture on a new session. Without the list, Explore cannot tell whether the session can switch appearance, approve the app's links, or restart SpringBoard.",
details: { sessionId, attempts: retry + 1 } });
// A list that succeeds on a later attempt would otherwise leave no trace of the failure in the job log.
process.stderr.write(`simulator:control --list failed for session ${sessionId} (${failure}); asking again in ${delayMs / 1000} s\n`);
sleep(delayMs);
}
}
/** Lets a test replace the pause between resends of a failed action. */
export type SimulatorControlOptions = { sleep?: ((milliseconds: number) => void) | undefined };
+8 -5
View File
@@ -15,7 +15,10 @@ import { ARGENT_VERSION } from "../shared/argent-version.js";
const METADATA = "cache-metadata.json";
const CONTRACT = "explore.capture-cache";
const MAX_BYTES = 250 * 1024 * 1024;
/** The most evidence a capture cache holds; a downloaded baseline is refused past it before it is unpacked. */
export const CAPTURE_CACHE_MAX_BYTES = 250 * 1024 * 1024;
/** The most files a capture cache holds, besides its metadata. */
export const CAPTURE_CACHE_MAX_FILES = 10_000;
type Recipe = { inventory: unknown; profile: string; config: string; toolsCommit: string };
type FileRecord = { path: string; bytes: number; sha256: string };
type CacheMetadata = { contract: typeof CONTRACT; recipe: string; commit: string; files: FileRecord[] };
@@ -28,9 +31,9 @@ const recipeKey = ({ inventory, profile, config, toolsCommit }: Recipe) => {
/** Only settings that can change captured runtime evidence belong in the recipe. */
export function captureRecipeConfig(config: ExploreConfig, instructionsSha256?: string): string {
// How many sessions capture the evidence (capture.maxSessions) does not change it.
const { maxSessions: _maxSessions, ...capture } = config.capture;
void _maxSessions;
// How many sessions capture the evidence (capture.maxSessions), and how long a review waits for a baseline, do not change it.
const { maxSessions: _maxSessions, baselineWaitMinutes: _baselineWaitMinutes, ...capture } = config.capture;
void _maxSessions; void _baselineWaitMinutes;
return canonicalJson({
observer: CAPTURE_OBSERVER_HASH,
mode: config.mode,
@@ -104,7 +107,7 @@ async function fileInventory(directory: string, path = ""): Promise<FileRecord[]
else if (info.isFile()) files.push({ path: child, bytes: info.size, sha256: hash(await readFile(join(directory, child))) });
else throw new Error("Capture cache contains an unsupported file");
}
if (files.length > 10_000 || files.reduce((sum, file) => sum + file.bytes, 0) > MAX_BYTES) {
if (files.length > CAPTURE_CACHE_MAX_FILES || files.reduce((sum, file) => sum + file.bytes, 0) > CAPTURE_CACHE_MAX_BYTES) {
throw new Error("Capture cache exceeds the evidence size limit");
}
return files;
+203
View File
@@ -0,0 +1,203 @@
import { openAsBlob } from "node:fs";
import { mkdir, mkdtemp, open, rm, writeFile } from "node:fs/promises";
import { dirname, join, relative, resolve } from "node:path";
import { setTimeout as delay } from "node:timers/promises";
import { BlobReader, Uint8ArrayWriter, ZipReader, configure } from "@zip.js/zip.js";
import { CiError } from "../shared/ci-errors.js";
import { silentProgress, type Progress } from "../shared/progress.js";
import { retryGitHubLookup } from "../report/github-lookup.js";
import { safeRelativePath } from "../binary/archive.js";
import { CAPTURE_CACHE_MAX_BYTES, CAPTURE_CACHE_MAX_FILES } from "./artifact-cache.js";
import { baselineLookup, githubContext, trustedBaselineRun, type GithubArtifact, type GithubRun } from "./workflow-steps.js";
/**
* A review that waits for main's baseline of its base commit instead of capturing the base itself. Right after a
* merge, the next pull request's review starts while the baseline workflow is still capturing the new base: in
* brentvatne/euxy's review run 39348956443 it had started 15 s before, so every target captured its base too, about
* 10–14 minutes per target instead of 6, on twice the EAS Simulator sessions. A baseline job takes 6–8 minutes and
* head capture 5–8, so `runCiReview` starts head at once and waits here only where base would start.
*/
/** `capture.baselineWaitMinutes` when the config sets none: two baseline jobs' worth, so one queued behind another still counts. */
export const DEFAULT_BASELINE_WAIT_MINUTES = 15;
const POLL_MS = 20_000;
/** GitHub's Actions API lists an artifact a little after its upload (#47); a finished job gets this long for its artifact to appear. */
const ARTIFACT_LISTING_GRACE_MS = 60_000;
/** A zip of the largest capture cache, stored rather than deflated, plus its headers. */
const MAX_ARTIFACT_ZIP_BYTES = CAPTURE_CACHE_MAX_BYTES + 16 * 1024 * 1024;
type GithubJob = { name: string; status?: string; conclusion?: string | null };
export type BaselineWaitOptions = {
/** The baseline run `ci:find-baseline` named; it is checked again here, by the same trust rules. */
run: number; sha: string; target: string;
/** The longest the review waits, from its first poll (`capture.baselineWaitMinutes`). */
timeoutMs: number; pollMs?: number;
environment?: NodeJS.ProcessEnv; transport?: typeof fetch; progress?: Progress; signal?: AbortSignal;
now?: () => number; sleep?: (milliseconds: number, signal?: AbortSignal) => Promise<void>;
};
/** A failed GitHub read that may succeed later: no answer or a 5xx. A bad token or a missing permission is not. */
const transientGitHubFailure = (error: unknown) => error instanceof CiError && error.retryable;
const describe = (error: unknown) => error instanceof CiError ? `${error.code}: ${error.message}` : error instanceof Error ? error.message : String(error);
const statusText = (status: string | undefined) => (status ?? "queued").replaceAll("_", " ");
/**
* Polls one baseline run until this target's baseline artifact is listed, and answers with it. The run must pass the
* trust check a finished baseline passes. Each target's job uploads its own artifact when it finishes, so a target is
* reused even when another target's job fails (#48); this target's job failing or being cancelled ends the wait, as do
* the time limit, a missing permission and GitHub failures that outlast their retries. The reason says which.
*/
export async function waitForBaselineArtifact(options: BaselineWaitOptions): Promise<{ artifact: { id: number; run: number } } | { reason: string }> {
const now = options.now ?? Date.now;
const sleep = options.sleep ?? ((milliseconds: number, signal?: AbortSignal) => delay(milliseconds, undefined, signal ? { signal } : {}));
const progress = options.progress ?? silentProgress;
const started = now();
const short = options.sha.slice(0, 8);
try {
const lookup = await baselineLookup({ sha: options.sha, target: options.target, ...(options.environment ? { environment: options.environment } : {}),
...(options.transport ? { transport: options.transport } : {}),
retry: read => retryGitHubLookup(read, transientGitHubFailure, milliseconds => sleep(milliseconds, options.signal)) });
if (!lookup) return { reason: "the repository has no baseline workflow" };
const { context, get, target } = lookup;
// The matrix names each baseline job after its target; an older single-platform workflow's one job is `baseline`.
const jobNames = [`baseline (${target})`, ...(target === "ios" ? ["baseline"] : [])];
let finishedAt: number | undefined;
let logged = "";
for (;;) {
const run = await get<GithubRun>(`repos/${context.repository}/actions/runs/${options.run}`);
if (!trustedBaselineRun(run, lookup)) return { reason: `run ${options.run} is not a baseline run of ${short} on ${lookup.defaultBranch}` };
for (const name of lookup.names) {
const listed = await get<{ artifacts?: GithubArtifact[] }>(`repos/${context.repository}/actions/runs/${run.id}/artifacts?name=${encodeURIComponent(name)}&per_page=100`);
const artifact = (listed?.artifacts ?? []).filter(candidate => !candidate.expired && (candidate.name ?? name) === name
&& Number.isSafeInteger(candidate.id) && candidate.id > 0).sort((a, b) => b.id - a.id)[0];
if (artifact) return { artifact: { id: artifact.id, run: run.id } };
}
const jobs = await get<{ jobs?: GithubJob[] }>(`repos/${context.repository}/actions/runs/${run.id}/jobs?per_page=100`);
const job = jobs?.jobs?.find(candidate => jobNames.includes(candidate.name));
if (job?.status === "completed" && job.conclusion !== "success") {
return { reason: `its ${target} job ${job.conclusion === "cancelled" ? "was cancelled" : job.conclusion === "failure" ? "failed" : `ended (${job.conclusion ?? "no conclusion"})`}` };
}
if (job?.status === "completed" || run.status === "completed") {
finishedAt ??= now();
if (now() - finishedAt >= ARTIFACT_LISTING_GRACE_MS) return { reason: job ? `its ${target} job finished, but GitHub lists no ${lookup.names[0]} artifact`
: `the run ended (${run.conclusion ?? "no conclusion"}) without a ${target} baseline job` };
}
const waited = now() - started;
const state = job ? statusText(job.status) : run.status === "completed" ? "finished" : "not started";
if (waited >= options.timeoutMs) return { reason: `its ${target} job is still ${state} after ${Math.round(waited / 60_000)} min` };
// A line when the state changes and once a minute, not on every poll.
const line = `waiting for ${lookup.defaultBranch}'s baseline run ${run.id} of ${short} (${target}: ${state}, ${Math.floor(waited / 60_000)} min)`;
if (line !== logged) { progress(line); logged = line; }
await sleep(Math.min(options.pollMs ?? POLL_MS, options.timeoutMs - waited), options.signal);
}
} catch (error) {
return { reason: options.signal?.aborted ? "the review stopped waiting" : describe(error) };
}
}
/**
* Downloads a baseline artifact and unpacks it into `directory`, which must not exist yet: what
* actions/download-artifact would have put in EXPLORE_CAPTURE_CACHE. The token goes to GitHub only, never to the
* storage URL GitHub redirects to. Paths are confined to the directory and the cache's size limits apply before
* anything is written; restoreCaptureArtifactCache then checks the evidence as it checks a downloaded one.
*/
export async function downloadBaselineArtifact(options: { artifact: number; directory: string; environment?: NodeJS.ProcessEnv; transport?: typeof fetch;
signal?: AbortSignal; sleep?: (milliseconds: number, signal?: AbortSignal) => Promise<void> }): Promise<void> {
const environment = options.environment ?? process.env; const transport = options.transport ?? fetch;
const context = githubContext(environment);
const sleep = options.sleep ?? ((milliseconds: number, signal?: AbortSignal) => delay(milliseconds, undefined, signal ? { signal } : {}));
const signal = (milliseconds: number) => options.signal ? AbortSignal.any([options.signal, AbortSignal.timeout(milliseconds)]) : AbortSignal.timeout(milliseconds);
const path = `repos/${context.repository}/actions/artifacts/${options.artifact}/zip`;
// A 404 just after the artifact was listed is the Actions API catching up (#47), as a 5xx or no answer may be.
const located = await retryGitHubLookup(async () => {
let response: Response;
try {
response = await transport(`https://api.github.com/${path}`, { redirect: "manual", signal: signal(60_000),
headers: { authorization: `Bearer ${context.token}`, accept: "application/vnd.github+json", "x-github-api-version": "2022-11-28" } });
} catch (error) {
throw new CiError({ code: "GITHUB_REQUEST_FAILED", stage: "capture-base", message: `GitHub could not be reached (${path}).`,
nextAction: "Check GitHub availability.", retryable: true, cause: error });
}
if (response.status >= 300 && response.status < 400) {
await response.body?.cancel().catch(() => {});
return { location: response.headers.get("location") ?? "" };
}
if (response.ok) return { response };
await response.body?.cancel().catch(() => {});
throw new CiError({ code: response.status === 403 ? "GITHUB_PERMISSION_DENIED" : "GITHUB_REQUEST_FAILED", stage: "capture-base",
message: `GitHub answered HTTP ${response.status} for ${path}.`, retryable: response.status === 404 || response.status >= 500,
nextAction: response.status === 403 ? "Grant the capture job actions: read." : "Rerun the workflow; check GitHub's status if it keeps failing." });
}, transientGitHubFailure, milliseconds => sleep(milliseconds, options.signal));
let response = located.response;
if (!response) {
const location = URL.canParse(located.location ?? "") ? new URL(located.location!) : undefined;
if (location?.protocol !== "https:") throw new Error("GitHub redirected the artifact download to no HTTPS address");
response = await transport(location.href, { signal: signal(10 * 60_000) });
if (!response.ok || !response.body) throw new Error(`The artifact download answered HTTP ${response.status}`);
}
await mkdir(dirname(resolve(options.directory)), { recursive: true });
const stage = await mkdtemp(`${resolve(options.directory)}.download-`);
try {
const archive = join(stage, "artifact.zip");
const file = await open(archive, "wx", 0o600);
try {
let bytes = 0;
for await (const chunk of response.body as unknown as AsyncIterable<Uint8Array>) {
bytes += chunk.byteLength;
if (bytes > MAX_ARTIFACT_ZIP_BYTES) throw new Error("The baseline artifact is larger than a capture cache can be");
await file.write(chunk);
}
} finally { await file.close(); }
await unpackArtifact(archive, options.directory);
} finally {
await rm(stage, { recursive: true, force: true });
}
}
async function unpackArtifact(archive: string, directory: string): Promise<void> {
configure({ useWebWorkers: false });
const zip = new ZipReader(new BlobReader(await openAsBlob(archive)), { useWebWorkers: false });
try {
const entries = await zip.getEntries();
const files = entries.filter(entry => !entry.directory);
// The cache's own metadata file is the one file beyond its evidence.
if (files.length > CAPTURE_CACHE_MAX_FILES + 1 || files.reduce((sum, entry) => sum + entry.uncompressedSize, 0) > CAPTURE_CACHE_MAX_BYTES + 2 * 1024 * 1024) {
throw new Error("The baseline artifact exceeds the capture cache's limits");
}
const root = resolve(directory);
await mkdir(root);
for (const entry of files) {
const path = safeRelativePath(entry.filename);
const target = path ? resolve(root, path) : undefined;
if (!target || relative(root, target).startsWith("..") || !entry.getData) throw new Error(`The baseline artifact holds an unsafe path: ${entry.filename}`);
await mkdir(dirname(target), { recursive: true });
await writeFile(target, await entry.getData(new Uint8ArrayWriter(), { checkSignature: true, useWebWorkers: false }), { flag: "wx", mode: 0o600 });
}
} finally {
await zip.close();
}
}
/**
* The wait as `capture:review` runs it: the downloaded baseline's directory, or undefined, and then base captures as it
* would have. It never fails the review; each outcome is one progress line.
*/
export async function awaitPendingBaseline(options: BaselineWaitOptions & { directory: string }): Promise<string | undefined> {
const progress = options.progress ?? silentProgress;
const short = options.sha.slice(0, 8);
const outcome = await waitForBaselineArtifact(options);
if ("reason" in outcome) {
progress(`base: stopped waiting for the baseline of ${short} (${outcome.reason}); capturing the base too`);
return undefined;
}
try {
await downloadBaselineArtifact({ ...options, artifact: outcome.artifact.id });
progress(`base: downloaded the baseline of ${short} that run ${outcome.artifact.run} captured`);
return options.directory;
} catch (error) {
await rm(options.directory, { recursive: true, force: true }).catch(() => {});
progress(`base: the baseline of ${short} from run ${outcome.artifact.run} could not be downloaded (${options.signal?.aborted ? "the review stopped waiting" : describe(error)}); capturing the base too`);
return undefined;
}
}
+69 -9
View File
@@ -18,6 +18,7 @@ import { quarantineReviewContract } from "../contracts/review-validation.js";
import { agentCaptureConstraints } from "../capture/agent/source-context.js";
import { captureArtifactCacheMayMatch, captureRecipeConfig, restoreCaptureArtifactCache, writeCaptureArtifactCache } from "./artifact-cache.js";
import { runCiMapOnWorker } from "./capture-worker.js";
import { DEFAULT_BASELINE_WAIT_MINUTES, awaitPendingBaseline } from "./baseline-wait.js";
import { writeCiReportInput, readAppDirectory } from "../report/artifact.js";
import { assessPublicEvidence, readExportedMap, type PublicEvidenceSide } from "../review/evidence-assessment.js";
import { parseRuntimeExecutionContext } from "../review/runtime-adapter.js";
@@ -173,6 +174,24 @@ export async function sideAppearancePasses(config: ExploreConfig, environment: N
return passes.length > 1 ? passes.map(pass => [...pass]) : undefined;
}
/**
* Base's start in a review that waits for a baseline still being captured; head has already started. Base captures as
* soon as the wait gives up, beside head when the sides capture concurrently, and a downloaded baseline is restored
* once head finishes, since restoring checks it against head's runtime profile. A head that fails ends the wait, and
* base is never started for a review that has already failed.
*/
export async function captureBaseAfterWait(input: { waited: Promise<string | undefined> | undefined; head: Promise<void>; stopWaiting: () => void;
captureBase: (cacheDirectory?: string) => Promise<void> }): Promise<void> {
if (!input.waited) return input.captureBase();
let headFailed = false;
input.head.catch(() => { headFailed = true; input.stopWaiting(); });
const waited = await input.waited;
if (headFailed) return;
if (!waited) return input.captureBase();
await input.head;
return input.captureBase(waited);
}
export function captureSidesConcurrently(input: { plan: SourceCapturePlan; agent: boolean; injectedRunner: boolean; baselineMayExist: boolean }): boolean {
return !input.agent && !input.injectedRunner && !input.baselineMayExist
&& input.plan.headCaptureSurfaceIds.length > 0 && input.plan.baseCaptureSurfaceIds.length > 0;
@@ -208,9 +227,30 @@ export type PublicCiOptions = {
runner?: CommandRunner;
/** Internal dependency injection; configuration cannot replace the authenticated CLI session. */
agentSessionFactory?: PrepareCiExplorationOptions["createSession"];
/** Internal dependency injection for tests: replaces the wait for a baseline run that EXPLORE_BASELINE_RUN names. */
awaitBaseline?: BaselineWait;
progress?: Progress;
};
/** Waits for a baseline of the base commit still being captured: its downloaded export's directory, or undefined. */
type BaselineWait = (signal: AbortSignal) => Promise<string | undefined>;
/**
* The wait for a baseline still capturing the base commit: for the run `ci:find-baseline` named in
* EXPLORE_BASELINE_RUN, which the wait checks again by the same trust rules, for up to capture.baselineWaitMinutes. A
* workflow that names no run, or a config that sets 0, captures the base as before.
*/
function pendingBaselineWait(options: Pick<PublicCiOptions, "awaitBaseline" | "config" | "target" | "platform" | "output" | "progress">,
environment: NodeJS.ProcessEnv, transport: typeof fetch, baseSha: string): BaselineWait | undefined {
if (options.awaitBaseline) return options.awaitBaseline;
const run = Number(environment.EXPLORE_BASELINE_RUN);
const minutes = options.config.capture.baselineWaitMinutes ?? DEFAULT_BASELINE_WAIT_MINUTES;
if (!Number.isSafeInteger(run) || run <= 0 || minutes === 0) return undefined;
const target = options.target ?? options.platform ?? options.config.target?.id ?? "ios";
return signal => awaitPendingBaseline({ run, sha: baseSha, target, timeoutMs: minutes * 60_000, directory: join(resolve(options.output), "baseline-download"),
environment, transport, signal, ...(options.progress ? { progress: options.progress } : {}) });
}
function assertRunIdentity(environment: NodeJS.ProcessEnv): void {
if (!/^[1-9][0-9]*$/.test(environment.GITHUB_RUN_ID ?? "") || !/^[1-9][0-9]*$/.test(environment.GITHUB_RUN_ATTEMPT ?? "")) {
throw new CiError({ code: "CI_RUN_IDENTITY_MISSING", stage: "preflight", message: "The GitHub run ID or attempt is missing.",
@@ -247,13 +287,14 @@ async function prepareCiRun(options: Pick<PublicCiOptions, "baseProject" | "head
}
/** A review of any two commits (`capture:review --compare FILE`): the same capture as a PR's, with no PR to verify or comment on. */
export async function runCompareCi(options: Omit<PublicCiOptions, "pullRequestPath" | "transport" | "onComment"> & { comparePath: string }) {
export async function runCompareCi(options: Omit<PublicCiOptions, "pullRequestPath" | "onComment"> & { comparePath: string }) {
const environment = options.environment ?? process.env;
assertRunIdentity(environment);
const comparison = projectComparePresentation(JSON.parse(await readFile(options.comparePath, "utf8")));
const { projects, output, state } = await prepareCiRun(options, environment, comparison);
const assertCurrentHead = async () => { for (const side of ["base", "head"] as const) assertCheckout(projects[side], comparison[side].sha); };
return runCiReview({ ...options, projects, output, state, presentation: comparison,
const awaitBaseline = pendingBaselineWait(options, environment, options.transport ?? fetch, comparison.base.sha);
return runCiReview({ ...options, projects, output, state, presentation: comparison, ...(awaitBaseline ? { awaitBaseline } : {}),
environment, runId: environment.GITHUB_RUN_ID!, runAttempt: environment.GITHUB_RUN_ATTEMPT!,
assertCurrentHead, postComment: async () => {} });
}
@@ -324,7 +365,8 @@ export async function runPublicCi(options: PublicCiOptions) {
for (const side of ["base", "head"] as const) assertCheckout(projects[side], pr[side].sha);
await assertCurrentPr();
};
return runCiReview({ ...options, projects, output, state, presentation: pr,
const awaitBaseline = pendingBaselineWait(options, environment, transport, pr.base.sha);
return runCiReview({ ...options, projects, output, state, presentation: pr, ...(awaitBaseline ? { awaitBaseline } : {}),
environment, runId: environment.GITHUB_RUN_ID!, runAttempt: environment.GITHUB_RUN_ATTEMPT!,
assertCurrentHead, postComment, canPublish: () => !prVerificationFailed });
}
@@ -354,6 +396,11 @@ export async function runCiReview(options: {
now?: () => number; sleep?: (milliseconds: number) => void; runner?: CommandRunner; subscription?: boolean; useLocalEasSession?: boolean;
agentSessionFactory?: PrepareCiExplorationOptions["createSession"];
prepareCapture?: () => Promise<void>;
/**
* Waits for a baseline of the base commit that is still being captured, when no downloaded baseline matches. It runs
* alongside head, and base waits for it only where base would start capturing.
*/
awaitBaseline?: BaselineWait;
progress?: Progress;
/** Stops the captures between steps; model requests take it through agentSessionFactory. */
signal?: AbortSignal;
@@ -371,6 +418,8 @@ export async function runCiReview(options: {
let usagePhase: AgentUsagePhase = "preflight";
let preserveReport: (() => Promise<void>) | undefined;
let reportWritten = false;
// Stops a baseline wait still polling once the review no longer needs it: done, failed, or stopped.
const stopWaiting = new AbortController();
try {
progress(`planning the review of ${pr.head.sha.slice(0, 8)} against base ${pr.base.sha.slice(0, 8)} (scope ${config.exploration.scope})`);
const plan = await buildVerifiedPlan({ baseProject: projects.base, headProject: projects.head,
@@ -401,6 +450,10 @@ export async function runCiReview(options: {
// A baseline that may be restored needs the head runtime profile first; otherwise both sides capture at once.
const baselineMayExist = plan.baseCaptureSurfaceIds.length > 0 && await captureArtifactCacheMayMatch({ ...cacheRecipes.base,
...(environment.EXPLORE_CAPTURE_CACHE ? { sourceDirectory: environment.EXPLORE_CAPTURE_CACHE } : {}), commit: pr.base.sha });
// With no finished baseline to reuse, one still being captured is waited for: from now, alongside head, and base
// waits for it only where it would start. Head is never delayed by it (brentvatne/euxy's review run 39348956443).
const waitedBaseline = !baselineMayExist && plan.baseCaptureSurfaceIds.length > 0 && options.awaitBaseline
? options.awaitBaseline(options.signal ? AbortSignal.any([stopWaiting.signal, options.signal]) : stopWaiting.signal).catch(() => undefined) : undefined;
const share = targetSessionShare(config, options.targets, options.posedTargets);
// A share of one session captures the sides in turn.
const concurrent = captureSidesConcurrently({ plan, agent: !!exploration.agent, injectedRunner: !!options.runner, baselineMayExist }) && share >= 2;
@@ -415,13 +468,13 @@ export async function runCiReview(options: {
progress(`sessions: ${share} of ${config.capture.maxSessions ?? DEFAULT_MAX_SESSIONS} for this target; `
+ (split.length ? `${split.join(" and ")} ${split.length === 1 ? "captures its" : "capture their"} appearances in separate sessions`
: concurrent ? "one session per side" : "the sides in turn"));
const captureSide = async (side: "head" | "base") => {
const captureSide = async (side: "head" | "base", cacheDirectory = environment.EXPLORE_CAPTURE_CACHE) => {
const directory = join(output, `${side}-export`);
const selectedIds = plan[`${side}CaptureSurfaceIds`];
const cacheRecipe = cacheRecipes[side];
let reuseDirectory: string | undefined;
if (side === "base" && selectedIds.length && runtimeProfile && environment.EXPLORE_CAPTURE_CACHE) {
const restored = await restoreCaptureArtifactCache({ ...cacheRecipe, sourceDirectory: environment.EXPLORE_CAPTURE_CACHE,
if (side === "base" && selectedIds.length && runtimeProfile && cacheDirectory) {
const restored = await restoreCaptureArtifactCache({ ...cacheRecipe, sourceDirectory: cacheDirectory,
commit: pr.base.sha, selectedIds, directory: join(output, "base-cache"), runtimeProfile,
...(exploration.agent ? { captureContexts: exploration.agent.contexts } : {}) });
if (restored) reuseDirectory = join(output, "base-cache");
@@ -453,12 +506,17 @@ export async function runCiReview(options: {
}
await writeCaptureArtifactCache(directory, cacheRecipe);
};
const captureBase = (head: Promise<void>) => captureBaseAfterWait({ waited: waitedBaseline, head, stopWaiting: () => stopWaiting.abort(),
captureBase: cacheDirectory => captureSide("base", cacheDirectory) });
if (waitedBaseline) progress(`base waits for the baseline run of ${pr.base.sha.slice(0, 8)} still being captured; head captures meanwhile`);
if (concurrent) {
stage = "capture-head"; state.phase = "capturing"; state.capture.current = "head"; usagePhase = "capture-head";
progress("capturing head and base at the same time in two simulator sessions");
progress(waitedBaseline ? "capturing head; base captures in a second simulator session if the baseline cannot be reused"
: "capturing head and base at the same time in two simulator sessions");
await postComment();
// Let both sides finish or fail on their own so every started session is stopped by its owner.
const settled = await Promise.allSettled([captureSide("head"), captureSide("base")]);
const head = captureSide("head");
const settled = await Promise.allSettled([head, captureBase(head)]);
for (const [index, outcome] of settled.entries()) {
if (outcome.status === "rejected") { stage = index === 0 ? "capture-head" : "capture-base"; throw outcome.reason; }
}
@@ -467,7 +525,7 @@ export async function runCiReview(options: {
stage = `capture-${side}`; state.phase = "capturing"; state.capture.current = side;
usagePhase = `capture-${side}`;
await postComment();
await captureSide(side);
await (side === "head" ? captureSide("head") : captureBase(Promise.resolve()));
}
}
delete state.capture.current;
@@ -564,6 +622,8 @@ export async function runCiReview(options: {
try { await postComment(); } catch { /* A stale PR or inaccessible GitHub must not replace the original failure. */ }
}
throw error;
} finally {
stopWaiting.abort();
}
}
+70 -21
View File
@@ -30,7 +30,7 @@ export async function writeCaptureOutputs(inputPath: string, environment: Enviro
await writeStepOutputs(environment, { preview: (await readCiReportInput(inputPath)).preview?.hosting ?? "" });
}
function githubContext(environment: Environment) {
export function githubContext(environment: Environment) {
const repository = environment.GITHUB_REPOSITORY ?? "";
const token = environment.GITHUB_TOKEN?.trim();
if (!/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/.test(repository)) throw new CiError({ code: "CI_RUN_IDENTITY_MISSING", stage: "preflight",
@@ -41,7 +41,7 @@ function githubContext(environment: Environment) {
}
/** One GitHub REST read; 404 is an answer (undefined), every other failure stops the step with what to do. */
async function githubGet<T>(path: string, context: { token: string }, transport: typeof fetch): Promise<T | undefined> {
export async function githubGet<T>(path: string, context: { token: string }, transport: typeof fetch): Promise<T | undefined> {
let response: Response;
try {
response = await transport(`https://api.github.com/${path}`, { redirect: "error", signal: AbortSignal.timeout(30_000),
@@ -102,60 +102,109 @@ async function trustedAuthors(context: { repository: string; token: string }, tr
catch (error) { return { logins: [], invalid: error instanceof Error ? error.message : String(error) }; }
}
type GithubArtifact = { id: number; expired: boolean; workflow_run?: { id?: number } };
type GithubRun = { conclusion: string | null; workflow_id: number; head_sha: string; head_branch: string; event: string; head_repository?: { full_name?: string } };
export type GithubArtifact = { id: number; name?: string; expired: boolean; workflow_run?: { id?: number } };
export type GithubRun = { id: number; status?: string; conclusion: string | null; workflow_id: number; head_sha: string; head_branch: string; event: string;
head_repository?: { full_name?: string } };
/**
* `ci:find-baseline`: the newest baseline-workflow artifact of exactly this commit, uploaded from the default branch of
* this repository by a finished run. Its id and run go to the step outputs for actions/download-artifact. A run that
* failed counts: a target's baseline job uploads its artifact only after every step before it passed, so another
* target's failure (one flaky simulator session, as in brentvatne/euxy#172) does not make this target's baseline
* unusable. A run still in progress or cancelled does not.
* What every baseline lookup starts from: the GitHub context, a checked commit and target, the default branch and the
* baseline workflow's id, and the artifact names a baseline of this target may carry. Undefined when the repository
* has no baseline workflow.
*/
export async function findBaselineArtifact(options: { sha: string; target?: string;
/** What callers from before targets pass: a platform is the id of its default target. */ platform?: string; environment?: Environment; transport?: typeof fetch }): Promise<{ id: number; run: number } | undefined> {
export async function baselineLookup(options: { sha: string; target?: string; platform?: string; environment?: Environment; transport?: typeof fetch;
/** Wraps each read, as a wait that polls for minutes retries GitHub's transient failures. */ retry?: <T>(read: () => Promise<T>) => Promise<T> }) {
const environment = options.environment ?? process.env; const transport = options.transport ?? fetch;
const context = githubContext(environment);
if (!SHA.test(options.sha)) throw new CiError({ code: "SHA_INVALID", stage: "preflight", message: "The base commit must be a full SHA.", nextAction: "Pass --sha with a 40-character commit SHA." });
const target = options.target ?? options.platform ?? "ios";
if (!isTargetId(target)) throw new CiError({ code: "TARGET_INVALID", stage: "preflight", message: `${target} is not a target id.`,
nextAction: "Pass --target with an id from targets: in .expo-explore/config.yaml (ios or android without targets:)." });
const repository = await githubGet<{ default_branch: string }>(`repos/${context.repository}`, context, transport);
const workflow = await githubGet<{ id: number }>(`repos/${context.repository}/actions/workflows/explore-baseline.yml`, context, transport);
const read = <T>(path: string) => githubGet<T>(path, context, transport);
const get = <T>(path: string) => options.retry ? options.retry(() => read<T>(path)) : read<T>(path);
const repository = await get<{ default_branch: string }>(`repos/${context.repository}`);
const workflow = await get<{ id: number }>(`repos/${context.repository}/actions/workflows/explore-baseline.yml`);
if (!repository || !workflow) return undefined;
// Baselines carry the target suffix; target ios also accepts the unsuffixed name older single-platform baseline workflows uploaded.
const names = [`explore-baseline-${options.sha}-${target}`, ...(target === "ios" ? [`explore-baseline-${options.sha}`] : [])];
return { context, get, target, sha: options.sha, defaultBranch: repository.default_branch, workflow: workflow.id, names };
}
/**
* Whether a run may supply this commit's baseline: a run of the baseline workflow for exactly this commit, on the
* default branch of this repository, started by a push or a manual dispatch. A finished run's artifact is reused, and
* a run still capturing is waited for, only when it passes this check; nothing from a pull request's head selects it.
*/
export function trustedBaselineRun(run: GithubRun | undefined, lookup: { workflow: number; sha: string; defaultBranch: string; context: { repository: string } }): run is GithubRun {
return !!run && run.workflow_id === lookup.workflow && run.head_sha === lookup.sha && run.head_branch === lookup.defaultBranch
&& ["push", "workflow_dispatch"].includes(run.event) && run.head_repository?.full_name === lookup.context.repository;
}
/**
* `ci:find-baseline`: the newest baseline-workflow artifact of exactly this commit, uploaded from the default branch of
* this repository by a finished run. Its id and run go to the step outputs for actions/download-artifact. A run that
* failed counts: a target's baseline job uploads its artifact only after every step before it passed, so another
* target's failure (one flaky simulator session, as in brentvatne/euxy#172) does not make this target's baseline
* unusable. A run still in progress or cancelled does not; findPendingBaselineRun finds one still capturing.
*/
export async function findBaselineArtifact(options: { sha: string; target?: string;
/** What callers from before targets pass: a platform is the id of its default target. */ platform?: string; environment?: Environment; transport?: typeof fetch }): Promise<{ id: number; run: number } | undefined> {
const lookup = await baselineLookup(options);
if (!lookup) return undefined;
const { context, get } = lookup;
const artifacts: GithubArtifact[] = [];
for (const name of names) for (let page = 1; page <= 10; page++) {
const listed = await githubGet<{ artifacts: GithubArtifact[] }>(`repos/${context.repository}/actions/artifacts?name=${encodeURIComponent(name)}&per_page=100&page=${page}`, context, transport);
for (const name of lookup.names) for (let page = 1; page <= 10; page++) {
const listed = await get<{ artifacts: GithubArtifact[] }>(`repos/${context.repository}/actions/artifacts?name=${encodeURIComponent(name)}&per_page=100&page=${page}`);
artifacts.push(...listed?.artifacts ?? []);
if ((listed?.artifacts.length ?? 0) < 100) break;
}
for (const artifact of artifacts.filter(candidate => !candidate.expired).sort((a, b) => b.id - a.id)) {
if (!artifact.workflow_run?.id) continue;
const run = await githubGet<GithubRun>(`repos/${context.repository}/actions/runs/${artifact.workflow_run.id}`, context, transport);
if (!run || (run.conclusion !== "success" && run.conclusion !== "failure") || run.workflow_id !== workflow.id || run.head_sha !== options.sha
|| run.head_branch !== repository.default_branch || !["push", "workflow_dispatch"].includes(run.event)
|| run.head_repository?.full_name !== context.repository) continue;
const run = await get<GithubRun>(`repos/${context.repository}/actions/runs/${artifact.workflow_run.id}`);
if (!trustedBaselineRun(run, lookup) || (run.conclusion !== "success" && run.conclusion !== "failure")) continue;
return { id: artifact.id, run: artifact.workflow_run.id };
}
return undefined;
}
/**
* A baseline run of exactly this commit that has not finished yet (queued, waiting or in progress), passing the same
* trust check as a finished one. In brentvatne/euxy's review run 39348956443 main's baseline of the base commit had
* started 15 s before the review, so no artifact existed yet. A run already capturing is preferred to one queued
* behind it in the baseline's concurrency group, since it finishes first; then the newest.
*/
export async function findPendingBaselineRun(options: Parameters<typeof baselineLookup>[0]): Promise<{ run: number } | undefined> {
const lookup = await baselineLookup(options);
if (!lookup) return undefined;
const listed = await lookup.get<{ workflow_runs?: GithubRun[] }>(`repos/${lookup.context.repository}/actions/workflows/${lookup.workflow}/runs?head_sha=${lookup.sha}`
+ `&branch=${encodeURIComponent(lookup.defaultBranch)}&per_page=100`);
const pending = (listed?.workflow_runs ?? []).filter(run => run.status !== "completed" && Number.isSafeInteger(run.id) && run.id > 0 && trustedBaselineRun(run, lookup))
.sort((a, b) => Number(b.status === "in_progress") - Number(a.status === "in_progress") || b.id - a.id);
return pending[0] ? { run: pending[0].id } : undefined;
}
/**
* `ci:find-baseline` as the workflow runs it: a baseline is optional, so a lookup that fails (a GitHub 5xx, a rate
* limit) is a warning and no baseline, never an error. A `ci:*` error would write explore-diagnostics/error.json,
* which the failure comment would then report as the cause of an unrelated failure, possibly on another target.
* With no finished baseline, it names a trusted baseline run still capturing the commit as `pending`, which
* `capture:review` waits for alongside head.
*/
export async function findOptionalBaseline(options: Parameters<typeof findBaselineArtifact>[0]):
Promise<{ baseline?: { id: number; run: number }; warning?: { code: string; message: string } }> {
Promise<{ baseline?: { id: number; run: number }; pending?: { run: number }; warning?: { code: string; message: string } }> {
try {
const baseline = await findBaselineArtifact(options);
return baseline ? { baseline } : {};
if (baseline) return { baseline };
} catch (error) {
return { warning: { code: error instanceof CiError ? error.code : "BASELINE_LOOKUP_FAILED",
message: `No baseline is reused, so capture captures the base too: ${error instanceof Error ? error.message : String(error)}` } };
}
try {
const pending = await findPendingBaselineRun(options);
return pending ? { pending } : {};
} catch (error) {
return { warning: { code: error instanceof CiError ? error.code : "BASELINE_LOOKUP_FAILED",
message: `No baseline run in progress is waited for, so capture captures the base too: ${error instanceof Error ? error.message : String(error)}` } };
}
}
/**
+10 -3
View File
@@ -110,13 +110,20 @@ export async function ciPullRequest(): Promise<void> {
output({ ok: true, pullRequest: presentation, author });
}
/** `ci:find-baseline --sha SHA [--platform P]`: a reusable baseline artifact of the base commit, if one exists. */
/**
* `ci:find-baseline --sha SHA [--platform P]`: a reusable baseline artifact of the base commit, if one exists, or else
* the baseline run still capturing it (`pending`), which `capture:review` waits for alongside head.
*/
export async function ciFindBaseline(): Promise<void> {
const platform = platformFlag(), target = targetFlag();
const { baseline, warning } = await findOptionalBaseline({ sha: required("sha"), ...(platform ? { platform } : {}), ...(target ? { target } : {}) });
const { baseline, pending, warning } = await findOptionalBaseline({ sha: required("sha"), ...(platform ? { platform } : {}), ...(target ? { target } : {}) });
if (baseline) await writeStepOutputs(process.env, { id: String(baseline.id), run: String(baseline.run) });
if (pending) {
await writeStepOutputs(process.env, { pending: String(pending.run) });
console.error(`No finished baseline of ${required("sha").slice(0, 8)} yet; baseline run ${pending.run} is still capturing it; capture:review waits for it while head captures, up to capture.baselineWaitMinutes.`);
}
if (warning && process.env.GITHUB_ACTIONS) console.log(`::warning title=${warning.code}::${warning.message.replace(/[\r\n]+/g, " ")}`);
output({ ok: true, baseline: baseline ?? null, ...(warning ? { warning } : {}) });
output({ ok: true, baseline: baseline ?? null, ...(pending ? { pending } : {}), ...(warning ? { warning } : {}) });
}
/** `ci:compare-refs --project DIR --base REF --head REF [--output FILE]`: the two commits a comparison captures. */
+1
View File
@@ -95,6 +95,7 @@ const CI = `CI (the installed workflows run these; they manage remote EAS Simula
(resolves the live PR and refuses it unless it is open, from this repository, and by a trusted author)
ci:compare-refs --project DIRECTORY --base REF --head REF [--output FILE]
ci:find-baseline --sha COMMIT [--target ID | --platform ios|android]
(a finished baseline's artifact of the commit, or else the default branch's baseline run still capturing it)
ci:after-capture --output TEMP_DIRECTORY --checkout DIRECTORY...
(stops the capture's EAS Simulator sessions and checks capture left each checkout unchanged)
(the ci:* commands write their results to $GITHUB_OUTPUT for the next steps)`;
+12 -1
View File
@@ -99,6 +99,11 @@ export type ExploreConfig = {
* DEFAULT_MAX_SESSIONS (5). Below the number of targets, that many target jobs run at once, one session each.
*/
maxSessions?: number;
/**
* How many minutes a review waits for the default branch's baseline run of its base commit, still capturing when
* the review starts, before it captures the base itself; 0 never waits. Absent: DEFAULT_BASELINE_WAIT_MINUTES (15).
*/
baselineWaitMinutes?: number;
/** Where the route observer is bundled into the app, when the config chooses; captureRepack gives the default. */
repack?: "eas" | "local"; appearance: AppearanceSetting; launch: LaunchSetting; reset: RouteResetSetting; unattributed: UnattributedSetting; readinessSeconds?: number;
/** The device controller; absent means agent-device, so configs that never chose one keep their cache recipes. */
@@ -194,6 +199,11 @@ function positiveInteger(value: unknown, label: string, fallback?: number): numb
return value;
}
function boundedInteger(value: unknown, label: string, minimum: number, maximum: number): number {
if (typeof value !== "number" || !Number.isSafeInteger(value) || value < minimum || value > maximum) fail(`${label} must be a whole number from ${minimum} to ${maximum}`, label);
return value;
}
function customScheme(value: unknown, label: string): string {
const result = text(value, label, 128).toLowerCase();
if (!/^[a-z][a-z0-9+.-]*$/.test(result) || RESERVED_SCHEMES.has(result)) {
@@ -300,7 +310,7 @@ export function validateExploreConfig(value: unknown): ExploreConfig {
const exploration = optionalRecord(root.exploration, "exploration");
exactKeys(exploration, ["scope", "exclude"], "exploration");
const capture = optionalRecord(root.capture, "capture");
exactKeys(capture, ["maxMinutes", "maxActions", "maxSessions", "repack", "appearance", "launch", "reset", "unattributed", "readinessSeconds", "controller", "simulator"], "capture");
exactKeys(capture, ["maxMinutes", "maxActions", "maxSessions", "baselineWaitMinutes", "repack", "appearance", "launch", "reset", "unattributed", "readinessSeconds", "controller", "simulator"], "capture");
const controller = literal(capture.controller, ["agent-device", "argent"] as const, "capture.controller", "agent-device");
const simulator = literal(capture.simulator, ["eas", "local"] as const, "capture.simulator", "eas");
const mode = literal(root.mode, ["deterministic", "agent"] as const, "mode", "deterministic");
@@ -348,6 +358,7 @@ export function validateExploreConfig(value: unknown): ExploreConfig {
maxMinutes: positiveInteger(capture.maxMinutes, "capture.maxMinutes", 30),
maxActions: positiveInteger(capture.maxActions, "capture.maxActions", 500),
...(capture.maxSessions === undefined ? {} : { maxSessions: positiveInteger(capture.maxSessions, "capture.maxSessions", 2) }),
...(capture.baselineWaitMinutes === undefined ? {} : { baselineWaitMinutes: boundedInteger(capture.baselineWaitMinutes, "capture.baselineWaitMinutes", 0, 60) }),
...(capture.repack === undefined ? {} : { repack: literal(capture.repack, ["eas", "local"] as const, "capture.repack") }),
appearance: parseAppearance(capture.appearance, "capture.appearance"),
launch: parseLaunchSetting(capture.launch, "capture.launch"),
+7
View File
@@ -143,6 +143,10 @@ jobs:
set -euo pipefail
npm ci --ignore-scripts --prefix "app-base/$APP_INSTALL_DIR"
npm ci --ignore-scripts --prefix "app-head/$APP_INSTALL_DIR"
# A finished baseline of the base commit is downloaded here. With none, ci:find-baseline names a baseline run still
# capturing it (pending): right after a merge the next review starts while main's baseline runs. Capture review
# checks that run again (this repository's baseline workflow, the default branch, a push or manual dispatch, exactly
# the base commit) and waits for its artifact while head captures, for up to capture.baselineWaitMinutes.
- name: Find optional exact-base evidence
id: cache
continue-on-error: true
@@ -164,6 +168,8 @@ jobs:
id: capture
env:
COMPARE: ${{ needs.targets.outputs.compare }}
# Reads the pending baseline run's jobs and artifact (actions: read); a comparison makes no other GitHub request.
GITHUB_TOKEN: ${{ github.token }}
SCOPE: ${{ inputs.scope }}
EXPIRES_IN: ${{ inputs.expires-in }}
EXPO_TOKEN: ${{ secrets.EXPO_TOKEN }}
@@ -172,6 +178,7 @@ jobs:
ANTHROPIC_WORKSPACE_ID: ${{ vars.ANTHROPIC_WORKSPACE_ID }}
# App-test secrets: one NAME: line per {{secret:NAME}} input in .expo-explore (expo-explore setup adds them).
EXPLORE_CAPTURE_CACHE: ${{ runner.temp }}/explore-cache
EXPLORE_BASELINE_RUN: ${{ steps.cache.outputs.pending }}
shell: bash
run: |
set -euo pipefail
+5
View File
@@ -133,6 +133,10 @@ jobs:
set -euo pipefail
npm ci --ignore-scripts --prefix "app-base/$APP_INSTALL_DIR"
npm ci --ignore-scripts --prefix "app-head/$APP_INSTALL_DIR"
# A finished baseline of the base commit is downloaded here. With none, ci:find-baseline names a baseline run still
# capturing it (pending): right after a merge the next review starts while main's baseline runs. Capture review
# checks that run again (this repository's baseline workflow, the default branch, a push or manual dispatch, exactly
# the base commit) and waits for its artifact while head captures, for up to capture.baselineWaitMinutes.
- name: Find optional exact-base evidence
id: cache
continue-on-error: true
@@ -161,6 +165,7 @@ jobs:
ANTHROPIC_WORKSPACE_ID: ${{ vars.ANTHROPIC_WORKSPACE_ID }}
# App-test secrets: one NAME: line per {{secret:NAME}} input in .expo-explore (expo-explore setup adds them).
EXPLORE_CAPTURE_CACHE: ${{ runner.temp }}/explore-cache
EXPLORE_BASELINE_RUN: ${{ steps.cache.outputs.pending }}
shell: bash
run: |
set -euo pipefail
+2
View File
@@ -169,6 +169,8 @@ const Navigation=createStaticNavigation(Root); export default function App(){ret
}
if (command === 'simulator:get') { const session = sessions.get(value(request, '--id')); return json({ id: session.id, name: session.name, type: 'agent-device', platform: platform.toUpperCase(), status: 'IN_PROGRESS', app: { id: PROJECT_ID, slug: 'example', ownerAccount: { name: 'acme' } }, remoteConfig: { agentDeviceRemoteSessionUrl: `https://controller.example/${session.id}`, agentDeviceRemoteSessionToken: 'controller-private-token' } }); }
if (command === 'simulator:stop') return json({ id: value(request, '--id'), status: 'STOPPED' });
// The session's serve-sim offers no simulator:control actions, so the capture falls back to the controller's own.
if (command === 'simulator:control' && request.args.includes('--list')) return json({});
}
if (basename(request.command) === 'repack-app') {
repackedSha = git(request.cwd, 'rev-parse', 'HEAD');
+18
View File
@@ -403,6 +403,24 @@ test("review captures and reports each configured target in its own matrix job,
}
});
test("a review waits for the baseline run ci:find-baseline names, with the token's Actions read and nothing the PR head chooses", () => {
for (const workflow of [review, compare]) {
const steps = workflow.jobs.capture.steps;
const find = steps.find(step => step.name === "Find optional exact-base evidence");
const capture = steps.find(step => step.name === "Capture review");
assert.equal(find["continue-on-error"], true, "a failed lookup never fails the review");
assert.ok(steps.indexOf(find) < steps.indexOf(capture));
// The run is the find step's output, chosen by the base commit and the default branch's baseline workflow; capture
// checks it again before it waits.
assert.equal(capture.env.EXPLORE_BASELINE_RUN, "${{ steps.cache.outputs.pending }}");
assert.equal(capture.env.GITHUB_TOKEN, "${{ github.token }}", "the wait reads the run, its jobs and its artifact");
assert.equal(workflow.jobs.capture.permissions.actions, "read");
assert.equal(workflow.jobs.capture.permissions.contents, "read");
}
assert.deepEqual(compare.jobs.capture.permissions, { contents: "read", actions: "read" }, "the wait needs no permission a comparison did not have");
assert.deepEqual(review.jobs.capture.permissions, { contents: "read", actions: "read", "pull-requests": "write" });
});
test("compare reviews any two refs from a manual run or another workflow, with no pull request access", () => {
for (const trigger of ["workflow_dispatch", "workflow_call"]) {
assert.deepEqual(Object.keys(compare.on[trigger].inputs).sort(), ["base", "expires-in", "head", "scope"]);
+120
View File
@@ -0,0 +1,120 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import { readFile, readdir } from 'node:fs/promises';
import { join } from 'node:path';
import { Uint8ArrayReader, Uint8ArrayWriter, ZipWriter, configure } from '@zip.js/zip.js';
import { runPublicCi, runBaselineCi, captureBaseAfterWait } from '../../dist/ci/pipeline.js';
import { readCaptureBundle } from '../../dist/review/capture-review.js';
import { fixture } from '../helpers/public-ci-fixture.mjs';
const deferred = () => { let resolve, reject; const promise = new Promise((done, fail) => { resolve = done; reject = fail; }); return { promise, resolve, reject }; };
const tick = () => new Promise(done => setImmediate(done));
/** The files of a baseline export, zipped as actions/upload-artifact uploads them. */
async function zipDirectory(directory) {
configure({ useWebWorkers: false });
const writer = new ZipWriter(new Uint8ArrayWriter(), { useWebWorkers: false });
for (const entry of await readdir(directory, { recursive: true, withFileTypes: true })) {
if (!entry.isFile()) continue;
const path = join(entry.parentPath, entry.name);
await writer.add(path.slice(directory.length + 1), new Uint8ArrayReader(await readFile(path)));
}
return writer.close();
}
test('base starts the moment the wait gives up, beside head, and a waited baseline is restored only once head has finished', async () => {
const events = [];
const run = ({ head, waited }) => captureBaseAfterWait({ waited: waited?.promise, head: head.promise, stopWaiting: () => events.push('stop waiting'),
captureBase: async cache => { events.push(cache ? `base from ${cache}` : 'base captures'); } });
// The baseline job failed while head was still capturing: base captures at once, in its own session.
let head = deferred(), waited = deferred();
let base = run({ head, waited });
waited.resolve(undefined); await tick();
assert.deepEqual(events, ['base captures'], 'base did not wait for head');
head.resolve(); await base;
// The baseline arrived first: restoring it needs head's runtime profile, so base starts after head.
events.length = 0; head = deferred(); waited = deferred();
base = run({ head, waited });
waited.resolve('/runner/baseline-download'); await tick();
assert.deepEqual(events, []);
head.resolve(); await base;
assert.deepEqual(events, ['base from /runner/baseline-download']);
// Head failed while base still waited: the wait stops, and base never starts for a failed review.
events.length = 0; head = deferred(); waited = deferred();
base = run({ head, waited });
head.reject(new Error('head capture failed')); await tick();
assert.deepEqual(events, ['stop waiting']);
waited.resolve(undefined); await base;
assert.deepEqual(events, ['stop waiting'], 'base was not captured');
// Without a wait, base starts as before.
events.length = 0;
await captureBaseAfterWait({ waited: undefined, head: new Promise(() => {}), stopWaiting: () => events.push('stop waiting'), captureBase: async () => { events.push('base captures'); } });
assert.deepEqual(events, ['base captures']);
});
test('head capture starts while the review waits for the baseline, and the waited baseline is restored for base', async t => {
const f = await fixture(t, 'expo-router');
t.mock.method(globalThis, 'fetch', async input => assert.fail(`Unmocked network request: ${input}`));
const baseline = await runBaselineCi({ project: f.baseProject, sha: f.baseSha, output: join(f.root, 'baseline'), config: f.config, environment: f.environment, runner: f.runner, now: f.now, sleep: f.sleep });
const before = f.requests.length;
// The baseline finishes only once head capture has started a simulator session; were head held back by the wait,
// the fallback would end it with no baseline after 20 s, and base would capture.
const finished = deferred();
let finishedBy;
const fallback = setTimeout(() => { finishedBy ??= 'timeout'; finished.resolve(undefined); }, 20_000);
t.after(() => clearTimeout(fallback));
const runner = request => {
if (request.args[0] === 'simulator:start' && !finishedBy) { finishedBy = 'head started'; setImmediate(() => finished.resolve(baseline.directory)); }
return f.runner(request);
};
let waitSignal;
const progress = [];
const output = join(f.root, 'review');
await runPublicCi({ baseProject: f.baseProject, headProject: f.headProject, pullRequestPath: f.pullRequestPath, output, config: f.config,
environment: f.environment, transport: f.transport, runner, now: f.now, sleep: f.sleep, progress: line => progress.push(line),
awaitBaseline: signal => { waitSignal = signal; return finished.promise; } });
assert.equal(finishedBy, 'head started');
assert.equal(f.requests.slice(before).filter(request => request.args[0] === 'simulator:start').length, 1, 'base reused the baseline instead of a second session');
assert.ok(progress.includes(`base waits for the baseline run of ${f.baseSha.slice(0, 8)} still being captured; head captures meanwhile`));
assert.ok(progress.includes(`base: found the baseline captures of ${f.baseSha.slice(0, 8)}`));
const baselineBundle = await readCaptureBundle(baseline.directory);
for (const [id, image] of Object.entries((await readCaptureBundle(join(output, 'base-export'))).images)) assert.deepEqual(image, baselineBundle.images[id]);
assert.equal(JSON.parse(await readFile(join(output, 'execution.json'), 'utf8')).capture.baseFromCache, true);
assert.equal(waitSignal.aborted, true, 'the wait is told to stop once the review is done');
});
test('capture:review waits for the baseline run ci:find-baseline named, downloads its artifact, and restores base from it', async t => {
const f = await fixture(t, 'expo-router');
t.mock.method(globalThis, 'fetch', async input => assert.fail(`Unmocked network request: ${input}`));
const baseline = await runBaselineCi({ project: f.baseProject, sha: f.baseSha, output: join(f.root, 'baseline'), config: f.config, environment: f.environment, runner: f.runner, now: f.now, sleep: f.sleep });
const zip = await zipDirectory(baseline.directory);
const actions = [];
const transport = async (input, init = {}) => {
const url = new URL(input);
if (url.origin === 'https://blob.example') { actions.push('download'); assert.equal(init.headers?.authorization, undefined); return new Response(zip); }
const path = url.pathname.replace('/repos/acme/example', '');
if (path === '' || path.startsWith('/actions/runs/40') || path.startsWith('/actions/workflows/') || path.startsWith('/actions/artifacts/77')) actions.push(path);
if (path === '') return Response.json({ default_branch: 'main' });
if (path === '/actions/workflows/explore-baseline.yml') return Response.json({ id: 5 });
if (path === '/actions/runs/40') return Response.json({ id: 40, status: 'in_progress', conclusion: null, workflow_id: 5, head_sha: f.baseSha, head_branch: 'main', event: 'push', head_repository: { full_name: 'acme/example' } });
if (path === '/actions/runs/40/artifacts') return Response.json({ artifacts: [{ id: 77, name: url.searchParams.get('name'), expired: false, workflow_run: { id: 40 } }] });
if (path === '/actions/artifacts/77/zip') return new Response(null, { status: 302, headers: { location: 'https://blob.example/77.zip' } });
return f.transport(input, init);
};
const review = async (name, config) => {
const before = f.requests.length; actions.length = 0;
const output = join(f.root, name);
const progress = [];
await runPublicCi({ baseProject: f.baseProject, headProject: f.headProject, pullRequestPath: f.pullRequestPath, output, config,
environment: { ...f.environment, EXPLORE_BASELINE_RUN: '40' }, transport, runner: f.runner, now: f.now, sleep: f.sleep, progress: line => progress.push(line) });
return { output, progress, sessions: f.requests.slice(before).filter(request => request.args[0] === 'simulator:start').length };
};
const waited = await review('review', f.config);
assert.deepEqual(actions, ['', '/actions/workflows/explore-baseline.yml', '/actions/runs/40', '/actions/runs/40/artifacts', '/actions/artifacts/77/zip', 'download']);
assert.ok(waited.progress.includes(`base: downloaded the baseline of ${f.baseSha.slice(0, 8)} that run 40 captured`));
assert.equal(waited.sessions, 1, 'base was restored from the downloaded baseline');
assert.equal(JSON.parse(await readFile(join(waited.output, 'execution.json'), 'utf8')).capture.baseFromCache, true);
const off = await review('review-no-wait', { ...f.config, capture: { ...f.config.capture, baselineWaitMinutes: 0 } });
assert.deepEqual(actions, [], 'capture.baselineWaitMinutes: 0 never waits');
assert.equal(off.sessions, 2, 'base captured itself');
});
+203
View File
@@ -0,0 +1,203 @@
import assert from "node:assert/strict";
import { mkdtemp, readFile, readdir, rm, stat } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
import { Uint8ArrayReader, Uint8ArrayWriter, ZipWriter, configure } from "@zip.js/zip.js";
const { waitForBaselineArtifact, awaitPendingBaseline } = await import("../../dist/ci/baseline-wait.js");
const { findPendingBaselineRun, findOptionalBaseline } = await import("../../dist/ci/workflow-steps.js");
const sha = "a".repeat(40);
const environment = { GITHUB_REPOSITORY: "acme/app", GITHUB_TOKEN: "test-token" };
const trustedRun = { id: 20, workflow_id: 77, head_sha: sha, head_branch: "main", event: "push", head_repository: { full_name: "acme/app" } };
const job = (status, conclusion = null, target = "iphone") => ({ name: `baseline (${target})`, status, conclusion });
async function zipFiles(files) {
configure({ useWebWorkers: false });
const writer = new ZipWriter(new Uint8ArrayWriter(), { useWebWorkers: false });
for (const [name, content] of Object.entries(files)) await writer.add(name, new Uint8ArrayReader(new TextEncoder().encode(content)));
return writer.close();
}
/**
* GitHub as a review's wait sees it: baseline run 20 of `sha`, whose state moves to the next of `states` each time the
* run is read (the last one stays), and artifact 500 of the iphone target, served from a storage URL GitHub redirects to.
*/
function fakeGitHub({ states = [{}], run = {}, zip, runFailures = [], runs } = {}) {
let poll = -1;
const requests = [];
const state = () => states[Math.min(Math.max(poll, 0), states.length - 1)];
const transport = async (input, init = {}) => {
const url = new URL(input);
requests.push(`${url.pathname}${url.search}`);
if (url.origin === "https://blob.example") {
assert.equal(init.headers?.authorization, undefined, "the token never reaches the storage URL GitHub redirects to");
return new Response(zip);
}
assert.equal(url.origin, "https://api.github.com");
assert.equal(init.headers?.authorization, "Bearer test-token");
const path = url.pathname;
if (path === "/repos/acme/app") return Response.json({ default_branch: "main" });
if (path === "/repos/acme/app/actions/workflows/explore-baseline.yml") return Response.json({ id: 77 });
if (path === "/repos/acme/app/actions/artifacts") return Response.json({ artifacts: [] });
if (path === "/repos/acme/app/actions/workflows/77/runs") {
assert.equal(url.searchParams.get("head_sha"), sha);
assert.equal(url.searchParams.get("branch"), "main");
return Response.json({ workflow_runs: runs ?? [] });
}
if (path === "/repos/acme/app/actions/runs/20") {
poll += 1;
const failure = runFailures.shift();
if (failure) return new Response("unavailable", { status: failure });
return Response.json({ ...trustedRun, status: state().runStatus ?? "in_progress", conclusion: state().conclusion ?? null, ...run });
}
if (path === "/repos/acme/app/actions/runs/20/artifacts") {
const name = url.searchParams.get("name");
return Response.json({ artifacts: state().artifact && name === `explore-baseline-${sha}-iphone` ? [{ id: 500, name, expired: false, workflow_run: { id: 20 } }] : [] });
}
if (path === "/repos/acme/app/actions/runs/20/jobs") return Response.json({ jobs: [{ name: "targets", status: "completed", conclusion: "success" }, ...state().jobs ?? []] });
if (path === "/repos/acme/app/actions/artifacts/500/zip") {
assert.equal(init.redirect, "manual", "the redirect is followed by hand, without the token");
return new Response(null, { status: 302, headers: { location: "https://blob.example/artifact.zip?sig=1" } });
}
throw new Error(`unexpected request ${url}`);
};
return { transport, requests, polls: () => poll + 1 };
}
/** A clock the wait's pauses advance, so fifteen minutes of polling take no real time. */
function clock() {
let milliseconds = 0;
return { now: () => milliseconds, sleep: async pause => { milliseconds += pause; }, elapsed: () => milliseconds };
}
const waitOptions = (github, time, extra = {}) => ({ run: 20, sha, target: "iphone", timeoutMs: 15 * 60_000, environment, transport: github.transport,
now: time.now, sleep: time.sleep, ...extra });
test("a baseline job in progress, then finished: the review reuses its artifact, downloaded and unpacked as download-artifact would", async t => {
const root = await mkdtemp(join(tmpdir(), "explore-baseline-wait-"));
t.after(() => rm(root, { recursive: true, force: true }));
const zip = await zipFiles({ "bundle.json": "{\"side\":\"head\"}", "cache-metadata.json": "{}", "screenshots/home.png": "png" });
const github = fakeGitHub({ zip, states: [{ jobs: [] }, { jobs: [job("queued")] }, { jobs: [job("in_progress")] }, { jobs: [job("in_progress")] },
{ jobs: [job("completed", "success")], artifact: true }] });
const time = clock(); const lines = [];
const directory = join(root, "review", "baseline-download");
assert.equal(await awaitPendingBaseline({ ...waitOptions(github, time, { progress: line => lines.push(line) }), directory }), directory);
assert.deepEqual((await readdir(directory, { recursive: true })).sort(), ["bundle.json", "cache-metadata.json", "screenshots", "screenshots/home.png"]);
assert.equal(await readFile(join(directory, "screenshots/home.png"), "utf8"), "png");
assert.equal(github.polls(), 5);
assert.deepEqual(lines, [
"waiting for main's baseline run 20 of aaaaaaaa (iphone: not started, 0 min)",
"waiting for main's baseline run 20 of aaaaaaaa (iphone: queued, 0 min)",
"waiting for main's baseline run 20 of aaaaaaaa (iphone: in progress, 0 min)",
"waiting for main's baseline run 20 of aaaaaaaa (iphone: in progress, 1 min)",
"base: downloaded the baseline of aaaaaaaa that run 20 captured",
]);
assert.ok(github.requests.every(request => !request.startsWith("/repos/acme/app/actions/runs/") || request.startsWith("/repos/acme/app/actions/runs/20")),
"only the run ci:find-baseline named is read");
assert.deepEqual((await readdir(join(root, "review"))).sort(), ["baseline-download"], "the downloaded zip is removed once unpacked");
});
test("this target's artifact is reused although another target's baseline job failed (#48)", async () => {
const github = fakeGitHub({ states: [{ jobs: [job("completed", "failure", "duo-open"), job("in_progress")] },
{ runStatus: "completed", conclusion: "failure", jobs: [job("completed", "failure", "duo-open"), job("completed", "success")], artifact: true }] });
assert.deepEqual(await waitForBaselineArtifact(waitOptions(github, clock())), { artifact: { id: 500, run: 20 } });
});
test("a baseline job that fails or is cancelled ends the wait, and the review captures the base itself", async t => {
for (const [conclusion, reason] of [["failure", "its iphone job failed"], ["cancelled", "its iphone job was cancelled"], ["skipped", "its iphone job ended (skipped)"]]) {
const github = fakeGitHub({ states: [{ jobs: [job("in_progress")] }, { jobs: [job("completed", conclusion)] }] });
assert.deepEqual(await waitForBaselineArtifact(waitOptions(github, clock())), { reason }, conclusion);
}
const root = await mkdtemp(join(tmpdir(), "explore-baseline-wait-"));
t.after(() => rm(root, { recursive: true, force: true }));
const github = fakeGitHub({ states: [{ jobs: [job("in_progress")] }, { jobs: [job("completed", "failure")] }] });
const lines = [];
assert.equal(await awaitPendingBaseline({ ...waitOptions(github, clock(), { progress: line => lines.push(line) }), directory: join(root, "baseline-download") }), undefined);
assert.equal(lines.at(-1), "base: stopped waiting for the baseline of aaaaaaaa (its iphone job failed); capturing the base too");
assert.ok(!github.requests.some(request => request.includes("/zip")), "nothing is downloaded");
await assert.rejects(stat(join(root, "baseline-download")), { code: "ENOENT" });
});
test("a run that ends without this target's job, or a job whose artifact is never listed, ends the wait after a minute's grace", async () => {
const ended = fakeGitHub({ states: [{ runStatus: "completed", conclusion: "failure", jobs: [] }] });
const time = clock();
assert.deepEqual(await waitForBaselineArtifact(waitOptions(ended, time)), { reason: "the run ended (failure) without a iphone baseline job" });
assert.equal(time.elapsed(), 60_000);
const unlisted = fakeGitHub({ states: [{ jobs: [job("completed", "success")] }] });
assert.deepEqual(await waitForBaselineArtifact(waitOptions(unlisted, clock())), { reason: `its iphone job finished, but GitHub lists no explore-baseline-${sha}-iphone artifact` });
});
test("the wait gives up at its time limit, polling every 20 s", async () => {
const github = fakeGitHub({ states: [{ jobs: [job("in_progress")] }] });
const time = clock(); const lines = [];
assert.deepEqual(await waitForBaselineArtifact(waitOptions(github, time, { progress: line => lines.push(line) })), { reason: "its iphone job is still in progress after 15 min" });
assert.equal(time.elapsed(), 15 * 60_000);
assert.equal(github.polls(), 15 * 3 + 1);
assert.equal(lines.length, 15, "one line a minute, not one a poll");
assert.deepEqual(await waitForBaselineArtifact(waitOptions(fakeGitHub({ states: [{ jobs: [job("in_progress")] }] }), clock(), { timeoutMs: 50_000, pollMs: 30_000 })),
{ reason: "its iphone job is still in progress after 1 min" }, "the last pause ends at the limit");
});
test("only a baseline run of exactly the base commit, on the default branch of this repository, from a push or dispatch, is waited for", async () => {
for (const mutation of [{ head_branch: "feature" }, { event: "pull_request_target" }, { event: "pull_request" }, { workflow_id: 99 },
{ head_sha: "b".repeat(40) }, { head_repository: { full_name: "fork/app" } }]) {
const github = fakeGitHub({ run: mutation, states: [{ jobs: [job("completed", "success")], artifact: true }] });
assert.deepEqual(await waitForBaselineArtifact(waitOptions(github, clock())), { reason: "run 20 is not a baseline run of aaaaaaaa on main" }, JSON.stringify(mutation));
assert.ok(!github.requests.some(request => request.startsWith("/repos/acme/app/actions/runs/20/")), "nothing of an untrusted run is read");
}
const dispatched = fakeGitHub({ run: { event: "workflow_dispatch" }, states: [{ artifact: true }] });
assert.deepEqual(await waitForBaselineArtifact(waitOptions(dispatched, clock())), { artifact: { id: 500, run: 20 } });
});
test("ci:find-baseline names a trusted baseline run still capturing the base commit, preferring the one already capturing", async () => {
const run = (id, status, mutation = {}) => ({ ...trustedRun, id, status, conclusion: status === "completed" ? "success" : null, ...mutation });
const lookup = runs => findPendingBaselineRun({ sha, target: "iphone", environment, transport: fakeGitHub({ runs }).transport });
assert.deepEqual(await lookup([run(30, "completed"), run(31, "in_progress", { event: "pull_request_target" }), run(33, "queued"), run(32, "in_progress"),
run(34, "in_progress", { head_branch: "feature" }), run(35, "in_progress", { head_repository: { full_name: "fork/app" } })]), { run: 32 });
assert.deepEqual(await lookup([run(33, "queued"), run(36, "waiting")]), { run: 36 }, "otherwise the newest");
assert.equal(await lookup([run(30, "completed"), run(31, "in_progress", { workflow_id: 99 })]), undefined);
assert.deepEqual(await findOptionalBaseline({ sha, target: "iphone", environment, transport: fakeGitHub({ runs: [run(32, "in_progress")] }).transport }), { pending: { run: 32 } },
"with no finished baseline to download");
const outage = async input => new URL(input).pathname.endsWith("/77/runs") ? new Response("bad gateway", { status: 502 }) : fakeGitHub().transport(input, { headers: { authorization: "Bearer test-token" } });
const failed = await findOptionalBaseline({ sha, target: "iphone", environment, transport: outage });
assert.equal(failed.pending, undefined);
assert.match(failed.warning.message, /^No baseline run in progress is waited for, so capture captures the base too: GitHub answered HTTP 502/);
});
test("a GitHub 5xx while polling is retried, and a missing permission ends the wait without failing the review", async t => {
const github = fakeGitHub({ runFailures: [0, 502, 503], states: [{ jobs: [job("in_progress")] }, { jobs: [job("in_progress")] }, { jobs: [job("in_progress")] }, { artifact: true }] });
t.mock.method(process.stderr, "write", () => true);
assert.deepEqual(await waitForBaselineArtifact(waitOptions(github, clock())), { artifact: { id: 500, run: 20 } });
const outage = fakeGitHub({ runFailures: [502, 502, 502, 502, 502] });
assert.match((await waitForBaselineArtifact(waitOptions(outage, clock()))).reason, /^GITHUB_REQUEST_FAILED: GitHub answered HTTP 502/, "an outage that outlasts its retries ends the wait");
const denied = fakeGitHub({ runFailures: [403] });
assert.deepEqual(await waitForBaselineArtifact(waitOptions(denied, clock())),
{ reason: "GITHUB_PERMISSION_DENIED: GitHub answered HTTP 403 for repos/acme/app/actions/runs/20." });
assert.equal(denied.polls(), 1, "a missing permission is not retried");
const tokenless = await waitForBaselineArtifact(waitOptions(fakeGitHub(), clock(), { environment: { GITHUB_REPOSITORY: "acme/app" } }));
assert.match(tokenless.reason, /^GITHUB_TOKEN_MISSING/);
});
test("an artifact with a path outside its directory is refused before anything is written, and base captures as before", async t => {
const root = await mkdtemp(join(tmpdir(), "explore-baseline-wait-"));
t.after(() => rm(root, { recursive: true, force: true }));
const github = fakeGitHub({ zip: await zipFiles({ "bundle.json": "{}", "../escape.txt": "outside" }), states: [{ artifact: true }] });
const lines = [];
assert.equal(await awaitPendingBaseline({ ...waitOptions(github, clock(), { progress: line => lines.push(line) }), directory: join(root, "review", "baseline-download") }), undefined);
// zip.js refuses the entry as it reads the archive's directory, before the unpacking's own path check would.
assert.equal(lines.at(-1), "base: the baseline of aaaaaaaa from run 20 could not be downloaded (Unsafe filename); capturing the base too");
assert.deepEqual(await readdir(join(root, "review")), [], "neither the download nor a partial directory is left");
await assert.rejects(stat(join(root, "escape.txt")), { code: "ENOENT" });
});
test("a review that no longer needs the baseline stops waiting at once", async () => {
const github = fakeGitHub({ states: [{ jobs: [job("in_progress")] }] });
const controller = new AbortController();
const waiting = waitForBaselineArtifact({ run: 20, sha, target: "iphone", timeoutMs: 15 * 60_000, environment, transport: github.transport, signal: controller.signal });
setTimeout(() => controller.abort(), 50);
const started = Date.now();
assert.deepEqual(await waiting, { reason: "the review stopped waiting" });
assert.ok(Date.now() - started < 5_000);
});
@@ -138,6 +138,8 @@ test("capture recipes exclude display settings but bind agent instructions and c
// How many sessions capture it does not change the evidence: a baseline captured under any ceiling serves the review.
assert.equal(key,captureRecipeConfig({...config,capture:{...config.capture,maxSessions:16}}));
assert.equal(captureRecipeConfig({...config,capture:{...config.capture,maxSessions:4}}),captureRecipeConfig({...config,capture:{...config.capture,maxSessions:16}}));
// Nor does how long a review waits for a baseline still being captured.
assert.equal(key,captureRecipeConfig({...config,capture:{...config.capture,baselineWaitMinutes:0}}));
// An unset repack is keyed as it resolves, so configs that never chose one keep the recipe of the old written-out default.
assert.equal(key,captureRecipeConfig({...config,capture:{...config.capture,repack:"eas"}}));
assert.equal(JSON.parse(key).capture.repack, "eas");
+61 -1
View File
@@ -129,6 +129,8 @@ async function fixture(t, settings = {}) {
let enteredName = false; let scrolledToContinue = false; let finishHidden = false; let wedgeCleared = false; let deviceLost = false; let relaunchedAfterForeign = false; let serverStarts = 0; let unreadableLeft = settings.unreadableAfterFold ?? 0;
// startsErrored / startsUnnamed: how many session starts EAS fails, after naming the session (it ERRORS) or before.
let starts = 0; let currentSession = 'session-1'; const erroredSessions = new Set(); let foldUnverifiedLeft = settings.foldUnverified ?? 0;
// listFailures: the first so many `simulator:control --list` calls fail, as a host that drops the connection does.
let listFailuresLeft = settings.listFailures ?? 0;
// focusedField: the screen at `path` focuses its text field, and the keyboard is up from the `keyboardFrom`th read of it.
let focusedReads = 0;
// blackAfterFold: the first so many folds of an EAS agent-device session light the panel but draw nothing.
@@ -207,7 +209,10 @@ async function fixture(t, settings = {}) {
// controlActions: what the session's serve-sim offers through simulator:control (none unless a test names them).
if (command === 'simulator:control') {
assert.equal(value(request, '--id'), currentSession, 'simulator:control names the session');
if (request.args.includes('--list')) return json(Object.fromEntries((settings.controlActions ?? []).map(action => [action, { type: 'object' }])));
if (request.args.includes('--list')) {
if (listFailuresLeft > 0) { listFailuresLeft -= 1; return { stdout: '', stderr: 'The simulator preview server closed the connection before it answered.\nError: simulator:control command failed.\n' }; }
return json(Object.fromEntries((settings.controlActions ?? []).map(action => [action, { type: 'object' }])));
}
assert.ok((settings.controlActions ?? []).includes(request.args[1]), `${request.args[1]} is offered by the session`);
if (request.args[1] === 'appearance.set') appearance = request.args.at(-1);
return json({ id: currentSession, action: request.args[1], output: '' });
@@ -592,6 +597,61 @@ test('when a fold on EAS leaves Argent reading the dark panel, the capture resta
}
});
// In brentvatne/euxy's main baseline run 39487618529, one failed --list made the dark session look as if it offered no
// actions: it dropped dark, approved no link scheme, skipped the SpringBoard restart its fold needed, and failed the target.
test('a --list that fails once still gives an Argent dark session its appearance, its link approval and its fold recovery', async t => {
const f = await fixture(t, { argent: true, userInterfaceStyle: 'automatic', devicePose: 'open', foldUnverified: 1, listFailures: 1,
controlActions: ['appearance.set', 'app.approveUrlScheme', 'springboard.restart'] });
const lines = [];
const result = await runCiMap({ ...f.options, progress: line => lines.push(line), appearancePass: { all: ['light', 'dark'], capture: ['dark'] } });
assert.equal(result.runStatus, 'complete');
const control = f.requests.filter(request => request.args[0] === 'simulator:control');
assert.equal(control.filter(request => request.args.includes('--list')).length, 2);
assert.equal(lines.some(line => /Argent cannot switch appearance/.test(line)), false, lines.join('\n'));
assert.deepEqual(control.filter(request => request.args[1] === 'appearance.set').map(request => request.args.at(-1)), ['dark']);
assert.equal(control.filter(request => request.args[1] === 'app.approveUrlScheme').length, 1);
assert.equal(control.filter(request => request.args[1] === 'springboard.restart').length, 1);
const bundle = JSON.parse(await readFile(join(f.options.outputDirectory, 'bundle.json'), 'utf8'));
assert.deepEqual(Object.keys(bundle.images).sort(), [`${f.ids.home}@dark`, `${f.ids.settings}@dark`]);
});
test('a --list that keeps failing fails the session with a retryable error before the app is installed, and stops the session', async t => {
const f = await fixture(t, { argent: true, devicePose: 'open', listFailures: Infinity, controlActions: ['springboard.restart'] });
await assert.rejects(runCiMap(f.options), error => error.code === 'SIMULATOR_CONTROL_UNAVAILABLE' && error.retryable === true);
assert.equal(f.requests.filter(request => request.args[0] === 'simulator:control' && request.args.includes('--list')).length, 3);
assert.equal(f.requests.some(request => request.command.endsWith('/argent-controller') && ['install', 'fold'].includes(request.args[0])), false);
assert.equal(f.operations().filter(operation => operation === 'simulator:stop').length, 1);
});
test('a session given only dark that cannot switch to it stops before installing or folding, and the merge keeps the light session\'s capture', async t => {
const dark = await fixture(t, { argent: true, userInterfaceStyle: 'automatic', devicePose: 'open' });
const lines = [];
await runCiMap({ ...dark.options, progress: line => lines.push(line), appearancePass: { all: ['light', 'dark'], capture: ['dark'] } });
assert.equal(dark.requests.some(request => request.command.endsWith('/argent-controller') && ['install', 'fold'].includes(request.args[0])), false);
assert.equal(dark.operations().filter(operation => operation === 'simulator:stop').length, 1);
assert.ok(lines.some(line => /session session-1 cannot switch to dark, the appearance it was given; it stops without capturing/.test(line)), lines.join('\n'));
const idle = JSON.parse(await readFile(join(dark.options.outputDirectory, 'bundle.json'), 'utf8'));
assert.deepEqual(Object.keys(idle.images), []);
assert.equal(idle.outcomes[`${dark.ids.home}@dark`].error.code, 'SESSION_APPEARANCE_UNAVAILABLE');
// A light session that switched appearance expects dark: each route's dark frame is reported as not captured.
// One that could not switch either captured light alone, as one session would have: the dark session adds nothing.
for (const [actions, expected] of [[['appearance.set'], 'failed'], [[], undefined]]) {
const light = await fixture(t, { argent: true, userInterfaceStyle: 'automatic', devicePose: 'open', controlActions: actions });
await runCiMap({ ...light.options, appearancePass: { all: ['light', 'dark'], capture: ['light'] } });
await mergeCaptureExports(light.options.outputDirectory, [{ directory: dark.options.outputDirectory, scheme: 'dark' }]);
const merged = JSON.parse(await readFile(join(light.options.outputDirectory, 'bundle.json'), 'utf8'));
for (const id of Object.values(light.ids)) {
assert.equal(merged.outcomes[id].status, 'captured', id);
assert.equal(merged.outcomes[`${id}@dark`]?.status, expected, `${id}@dark with ${actions.join(', ') || 'no actions'}`);
}
}
// A session that cannot take the side's primary appearance fails it: the merge has nothing to add the others to.
const primary = await fixture(t, { argent: true, devicePose: 'open' });
await assert.rejects(runCiMap({ ...primary.options, appearance: ['dark', 'light'], appearancePass: { all: ['dark', 'light'], capture: ['dark'] } }),
error => error.code === 'SESSION_APPEARANCE_UNAVAILABLE');
assert.equal(primary.requests.some(request => request.command.endsWith('/argent-controller') && request.args[0] === 'fold'), false);
});
test('a session EAS ends while starting is replaced by a new one, under a name of its own, up to five times with a growing wait', async t => {
const f = await fixture(t, { startsErrored: 1 });
const lines = []; const waits = [];
+6
View File
@@ -47,6 +47,12 @@ test("rejects obsolete configuration and gives the exact next action", () => {
assert.equal(validateExploreConfig({ ...base(), capture: { readinessSeconds: 12 } }).capture.readinessSeconds, 12);
assert.equal("readinessSeconds" in validateExploreConfig(base()).capture, false);
assert.throws(() => validateExploreConfig({ ...base(), capture: { readinessSeconds: 0 } }), /capture.readinessSeconds must be a positive integer/);
// 0 never waits for a baseline still being captured; absent, the review waits DEFAULT_BASELINE_WAIT_MINUTES.
for (const minutes of [0, 15, 60]) assert.equal(validateExploreConfig({ ...base(), capture: { baselineWaitMinutes: minutes } }).capture.baselineWaitMinutes, minutes);
assert.equal("baselineWaitMinutes" in validateExploreConfig(base()).capture, false);
for (const minutes of [-1, 61, 1.5, "15"]) {
assert.throws(() => validateExploreConfig({ ...base(), capture: { baselineWaitMinutes: minutes } }), /capture.baselineWaitMinutes must be a whole number from 0 to 60/, String(minutes));
}
});
test("validates device, capture limits, providers, paths, and custom app schemes", () => {
+52 -3
View File
@@ -14,13 +14,62 @@ function easFake(listed) {
return { calls, eas };
}
test("a session's control actions come from --list, by session id; an older session or CLI has none", () => {
test("a session's control actions come from --list, by session id", () => {
const { calls, eas } = easFake(JSON.stringify({ "appearance.set": {}, "springboard.restart": {} }));
assert.deepEqual([...simulatorControlActions(eas, "session-1")], ["appearance.set", "springboard.restart"]);
assert.deepEqual(calls[0].args, ["simulator:control", "--list", "--id", "session-1", "--json", "--non-interactive"]);
assert.equal(calls[0].options.allowFailure, true);
assert.equal(simulatorControlActions(easFake("Unknown command simulator:control").eas, "session-1").size, 0);
assert.equal(simulatorControlActions(easFake(new Error("eas exited 1")).eas, "session-1").size, 0);
});
test("a session that lists no actions, or whose serve-sim or EAS CLI predates the list, has none, and is asked once", () => {
const pauses = [];
const options = { sleep: milliseconds => pauses.push(milliseconds) };
const empty = easFake("{}");
assert.equal(simulatorControlActions(empty.eas, "session-1", options).size, 0);
assert.equal(empty.calls.length, 1);
// eas-cli 25.2.0 on a session whose serve-sim refuses `actions.describe`, and an EAS CLI without simulator:control.
for (const reason of ["This simulator session runs a serve-sim version that does not support this action. Start a new session with `eas simulator:start` to get a newer version.",
"The simulator did not return its list of actions. Start a new session with `eas simulator:start` to get a newer serve-sim version."]) {
const calls = [];
const eas = args => { calls.push(args); return { stdout: "", stderr: `${reason}\nError: simulator:control command failed.\n`, status: 1 }; };
assert.equal(simulatorControlActions(eas, "session-1", options).size, 0);
assert.equal(calls.length, 1);
}
assert.equal(simulatorControlActions(easFake(new Error("eas exited with status 2: command simulator:control not found")).eas, "session-1", options).size, 0);
assert.deepEqual(pauses, []);
});
// One failed list made the dark session of euxy's main baseline run 39487618529 look as if it offered no actions.
function flakyList(failures, failure = () => ({ stdout: "", stderr: "The simulator preview server closed the connection before it answered.\nError: simulator:control command failed.\n", status: 1 })) {
const calls = [];
const eas = args => {
calls.push(args);
if (calls.length <= failures) return failure(calls.length);
return { stdout: JSON.stringify({ "appearance.set": {}, "app.approveUrlScheme": {}, "springboard.restart": {} }), stderr: "", status: 0 };
};
return { calls, eas };
}
test("a list that fails once is asked for again after a pause, and its actions are kept", () => {
for (const failure of [undefined, () => { throw new Error("eas simulator:control exited with status 1"); }, () => ({ stdout: "Connecting", stderr: "" })]) {
const { calls, eas } = flakyList(1, failure);
const pauses = [];
assert.deepEqual([...simulatorControlActions(eas, "session-1", { sleep: milliseconds => pauses.push(milliseconds) })], ["appearance.set", "app.approveUrlScheme", "springboard.restart"]);
assert.equal(calls.length, 2);
assert.deepEqual(calls[1], calls[0]);
assert.equal(pauses.length, 1);
}
});
test("a list that keeps failing throws a retryable error naming the session and EAS's reason, never an empty set", () => {
const { calls, eas } = flakyList(Infinity);
const pauses = [];
assert.throws(() => simulatorControlActions(eas, "session-1", { sleep: milliseconds => pauses.push(milliseconds) }), error =>
error.code === "SIMULATOR_CONTROL_UNAVAILABLE" && error.retryable === true && error.stage === "session"
&& /session-1/.test(error.message) && /closed the connection before it answered/.test(error.message) && !/command failed/.test(error.message));
assert.equal(calls.length, 3);
assert.equal(pauses.length, calls.length - 1);
assert.ok(pauses.every(milliseconds => milliseconds > 0 && milliseconds <= 10_000));
});
test("an action's values follow --, so one that starts with a dash is never read as a flag", () => {