mirror of
https://github.com/expo/explore.git
synced 2026-11-15 17:32:23 -10:00
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
edf0a60bb5 | ||
|
|
62940f5235 |
No files matched your search
+4
-4
@@ -153,10 +153,10 @@ Each new start logs the failed run's link and reason, for example `the hosted EA
|
||||
|
||||
1. `hostedRepackPeers` lists, in config order, the targets that repack the same app on EAS: one platform, EAS profile and bundle id. `capture:review`, `capture:review --compare` and `capture:baseline` pass them on.
|
||||
- A capture gets a `RepackShare` when it has two or more peers and `GITHUB_RUN_ID`, `GITHUB_RUN_ATTEMPT` and `GITHUB_REPOSITORY` are set. Its scope is the run attempt, its leader is the first peer, and its rank is its place among the peers.
|
||||
- A single target, a local review, `capture.repack: local`, and a native build this job made itself behave as before. Nobody else's key can match a native build this job made.
|
||||
2. The build to repack is the newest finished build with the fingerprint that Explore did not repack itself (`existingBuild`, from 20 builds). If repacks carry their source's fingerprint, every job still picks the same build.
|
||||
- A single target, a local review, `capture.repack: local`, and a native build this job made itself behave as before. The repack of a native build this job made carries no key, so it is never shared.
|
||||
2. The build to repack is the newest finished build with the fingerprint that Explore did not repack itself (`existingBuild`, from 20 builds), else the newest repack. On a project that reviews often, the 20 are all repacks (euxy's 50 most recent `explore` builds were), so jobs that start minutes apart can pick different builds; the key below names the native code, not the build, so they still share.
|
||||
3. Once the copy is staged, the capture takes a key (`hostedRepackShareKey`) from:
|
||||
- the run attempt, the platform and the native build id;
|
||||
- the run attempt, the platform and the native fingerprint;
|
||||
- the EAS profile and environment name;
|
||||
- a snapshot of the environment's variables: `eas env:list --format long` for the project and the account, which names each variable with its value (masked when sensitive or secret) and when it was last updated;
|
||||
- the Explore release;
|
||||
@@ -168,7 +168,7 @@ Each new start logs the failed run's link and reason, for example `the hosted EA
|
||||
6. When the repack it follows fails, is cancelled or loses the key, the capture looks for another with the same key for 60 s, staggered the same way, before it repacks itself. That catches the leader's retry, which starts at once (#176).
|
||||
7. It also repacks itself, with the retries above, when nothing appears in its wait, or when EAS fails to answer three times in a row. Looking stops while the budget still has 5 minutes for a repack of its own. A followed repack that does not finish within the budget fails the target. No capture waits on a job that cannot answer, and none waits past its budget.
|
||||
|
||||
Only a build whose marker is this capture's own key is used, and every input to the key is what this capture would itself have uploaded and built with. So two jobs share a repack only when their uploads, native build, profile, environment variables and Explore release are the same, in the same run attempt. The run attempt is in the scope, and a review's or a first attempt's repacks are never used by another run or attempt. That covers the baseline workflow, whose artifacts later reviews reuse, and "Re-run failed jobs" after an `EXPO_PUBLIC_*` value was fixed, which rebundles with the new value.
|
||||
Only a build whose marker is this capture's own key is used, and every input to the key but one is what this capture would itself have uploaded and built with. So two jobs share a repack only when their uploads, native fingerprint, profile, environment variables and Explore release are the same, in the same run attempt. The one input that is not this capture's own is the source build: a follower runs the leader's repack of the build the leader found, which has the same fingerprint as its own but may be another build of it. `existingBuild` already treats builds of one fingerprint as interchangeable; what a repack keeps from its source beyond the native code the fingerprint covers (its Info.plist, and the toolchain it was built with) can differ between such builds. The run attempt is in the scope, and a review's or a first attempt's repacks are never used by another run or attempt. That covers the baseline workflow, whose artifacts later reviews reuse, and "Re-run failed jobs" after an `EXPO_PUBLIC_*` value was fixed, which rebundles with the new value.
|
||||
|
||||
What the key does not protect against: every input to it is public or predictable, and the pull request's head code runs in the capture job with `EXPO_TOKEN` (`app.config.*` under `eas config` and `eas fingerprint:generate`, and the app's build code on EAS). Head code could therefore make a build whose message carries the base side's marker, and base captures of the same run attempt in other matrix jobs would install it. That is within the trust the capture already gives EAS: the same credentials could publish a simulator build with a reused fingerprint, which every capture already reuses by fingerprint (the `build` stage above), and head code in one job can already tamper with that job's own base side on the runner. The run attempt in the scope keeps such a build out of every other run, including the baseline. Requiring the build's `initiatingActor` to be this job's own credentials would not exclude head code, which runs with the same token. Explore cannot tell offline which actor EAS records for a workflow's repack build, so it does not check it.
|
||||
|
||||
|
||||
+17
-10
@@ -119,9 +119,9 @@ const EXPLORE_REPACK_MESSAGE = /^expo-explore (?:ios|android) capture: route obs
|
||||
|
||||
/**
|
||||
* The finished build to repack from `eas build:list --fingerprint-hash`: the newest one Explore did not repack itself,
|
||||
* else the newest. A repack keeps its source build's native code, and may carry its fingerprint; preferring the build
|
||||
* it came from has every job of a run pick the same one, which a shared repack's key needs (RepackShare), instead of a
|
||||
* repack another job finished a minute earlier.
|
||||
* else the newest. A repack keeps its source build's native code and carries its fingerprint, so on a project that
|
||||
* reviews often every listed build is a repack, and this is the newest of them; jobs that look minutes apart can get
|
||||
* different ones. A shared repack is keyed by the fingerprint, not by this build (hostedRepackShareKey).
|
||||
*/
|
||||
export function existingBuild(stdout: string): EasBuildDescriptor | undefined {
|
||||
const value = parseJson(stdout, "eas build:list");
|
||||
@@ -712,13 +712,18 @@ export async function stagedProjectDigest(root: string): Promise<string> {
|
||||
|
||||
/**
|
||||
* The key a shared repack's build carries in its message: everything the repack is made from, so two jobs share one only
|
||||
* when each would have uploaded the same bytes for the same native build, profile, EAS environment (by name and by a
|
||||
* snapshot of its variables) and Explore release, in the same run attempt.
|
||||
* when each would have uploaded the same bytes onto the same native code, for the same profile, EAS environment (by name
|
||||
* and by a snapshot of its variables) and Explore release, in the same run attempt.
|
||||
*
|
||||
* The native code is named by its fingerprint, not by the build a job found to repack: the build lookup returns the
|
||||
* newest build with that fingerprint, which is often an earlier repack, and it moves as repacks finish. In
|
||||
* brentvatne/euxy#179 (review run 40042718931) a target that started late found the repack the others had just made,
|
||||
* so its key matched none; it waited 3 minutes for a repack it could not find, then made another.
|
||||
*/
|
||||
export function hostedRepackShareKey(input: { scope: string; platform: "ios" | "android"; nativeBuildId: string; easProfile: string;
|
||||
export function hostedRepackShareKey(input: { scope: string; platform: "ios" | "android"; fingerprint: string; easProfile: string;
|
||||
environment: string; environmentVariables: string; project: string }): string {
|
||||
return createHash("sha256").update(JSON.stringify({ contract: "expo-explore.repack-share.v2", explore: EXPLORE_VERSION, scope: input.scope,
|
||||
platform: input.platform, nativeBuildId: input.nativeBuildId, easProfile: input.easProfile, environment: input.environment,
|
||||
return createHash("sha256").update(JSON.stringify({ contract: "expo-explore.repack-share.v3", explore: EXPLORE_VERSION, scope: input.scope,
|
||||
platform: input.platform, fingerprint: input.fingerprint, easProfile: input.easProfile, environment: input.environment,
|
||||
environmentVariables: input.environmentVariables, project: input.project })).digest("hex").slice(0, 32);
|
||||
}
|
||||
|
||||
@@ -870,6 +875,8 @@ async function prepareHostedRepack(options: {
|
||||
easCommand: string;
|
||||
project: string;
|
||||
nativeBuildId: string;
|
||||
/** The native fingerprint of that build, which keys a shared repack (hostedRepackShareKey). */
|
||||
fingerprint: string;
|
||||
easProfile: string;
|
||||
repackProfile: RepackProfile;
|
||||
easEnvironment: NodeJS.ProcessEnv;
|
||||
@@ -930,7 +937,7 @@ async function prepareHostedRepack(options: {
|
||||
if (variables === undefined) options.say(`EAS did not list the ${environment} environment's variables; repacking here without sharing the repack`);
|
||||
else {
|
||||
// The digest is taken before the workflow file is written: the file's message carries the key made from it.
|
||||
const marker = hostedRepackShareMarker(hostedRepackShareKey({ scope: options.share.scope, platform, nativeBuildId: options.nativeBuildId,
|
||||
const marker = hostedRepackShareMarker(hostedRepackShareKey({ scope: options.share.scope, platform, fingerprint: options.fingerprint,
|
||||
easProfile: options.easProfile, environment, environmentVariables: variables, project: await stagedProjectDigest(staged) }));
|
||||
const shared = await options.timings.measure("repack-share-wait", () => sharedHostedRepack({ share: options.share!, marker, platform, say: options.say, clock,
|
||||
list: () => eas(["build:list", "--platform", platform, "--limit", String(SHARED_REPACK_LIST_LIMIT), "--json", "--non-interactive"], 2 * 60_000).stdout,
|
||||
@@ -1012,7 +1019,7 @@ export async function prepareExactSourceBuild(options: {
|
||||
if ((options.repack ?? "eas") === "eas") {
|
||||
if (!options.easProfile) throw new Error("The hosted repack needs the capture build profile name.");
|
||||
const hosted = await prepareHostedRepack({ platform, runner: options.runner, easCommand: options.easCommand, project: options.project,
|
||||
nativeBuildId: options.nativeBuildId, easProfile: options.easProfile, repackProfile: options.repackProfile,
|
||||
nativeBuildId: options.nativeBuildId, fingerprint: options.fingerprint, easProfile: options.easProfile, repackProfile: options.repackProfile,
|
||||
easEnvironment: options.easEnvironment, restrictedEnvironment: options.repackEnvironment, provider: providers[0]!,
|
||||
parameterKeys: captureObserverParameterKeys(options.routes), root, share: options.share, say: options.progress ?? (() => undefined),
|
||||
timings: options.timings ?? silentTimings, onBlocking: options.onBlocking,
|
||||
|
||||
+1
-1
@@ -608,7 +608,7 @@ async function executeCiMap(options: CiMapOptions, setStage: (stage: string) =>
|
||||
type BuiltApp = { fingerprint: string; nativeEasBuildId: string | null; easBuildId: string | null; app?: string; reused: boolean; version?: string; buildNumber?: string };
|
||||
// What a hosted repack waits with and reports through: one deadline from the moment the session starts beside it, for
|
||||
// the wait for another job's repack and every repack of its own. The repack is shared with this run's other matrix
|
||||
// jobs only when it repacks a build EAS already had: a native build this job made is its own, so nobody else's key matches it.
|
||||
// jobs only when it repacks a build EAS already had: a native build this job made is its own, so its repack carries no key and is never shared.
|
||||
const repackWaits = (shared: boolean) => ({ ...(shared ? { share: options.repackShare } : {}), progress: say, now: options.now, sleep: options.sleep, timings, onBlocking: options.onBlocking,
|
||||
signal: options.signal, deadline: (options.now ?? Date.now)() + HOSTED_REPACK_BUDGET_MS });
|
||||
// `repacking` tells the captures sharing this build that their sessions can start too (sharedBuild).
|
||||
|
||||
@@ -243,7 +243,7 @@ async function fixture(t, settings = {}) {
|
||||
if (listed === null) throw new Error('eas env:list exited with status 1: GraphQL request failed.');
|
||||
return { stdout: listed ?? 'Environment: preview\nNo variables found for this environment.\n', stderr: '' };
|
||||
}
|
||||
if (command === 'build:list') return json(settings.reuse ? [{ id: 'native-build', ...buildMetadata }] : []);
|
||||
if (command === 'build:list') return json(settings.reuse ? [{ id: settings.reusable ?? 'native-build', ...buildMetadata }] : []);
|
||||
if (command === 'build') return json([{ id: 'built-app', fingerprint: { hash: 'source-fingerprint' }, ...buildMetadata }]);
|
||||
if (command === 'env:pull') { writeFileSync(value(request, '--path'), '# Environment: preview\nEXPO_PUBLIC_ENDPOINT=sample\n'); return { stdout: '', stderr: '' }; }
|
||||
if (command === 'build:download') {
|
||||
@@ -2146,6 +2146,22 @@ test('two jobs of one run attempt that repack the same app share one hosted repa
|
||||
assert.ok(lines.some(line => /using EAS build repacked-app, the hosted repack of this commit another target of this run made/.test(line)), lines.join('\n'));
|
||||
});
|
||||
|
||||
test('a target that starts after the shared repack finished, and finds that repack as its build to repack, joins it', async t => {
|
||||
const eas = easBuilds();
|
||||
const leader = await fixture(t, { reuse: true, eas });
|
||||
await runCiMap({ ...leader.options, ...share('iphone') });
|
||||
eas.builds[0].status = 'FINISHED';
|
||||
// The newest build with the fingerprint is now the leader's repack (brentvatne/euxy#179, review run 40042718931): a
|
||||
// key made from the build each job found would differ, and this target would wait, then repack a repack itself.
|
||||
const { f, lines, run } = await follow(t, eas, { reusable: 'repacked-app' });
|
||||
const result = await run;
|
||||
assert.equal(runs(f), 0, 'the late target does not repack');
|
||||
assert.equal(shareLists(f), 1, 'its first look finds the finished repack');
|
||||
assert.deepEqual(downloaded(f), ['repacked-app']);
|
||||
assert.equal(result.easBuildId, 'repacked-app');
|
||||
assert.ok(lines.some(line => /using EAS build repacked-app, the hosted repack of this commit another target of this run made/.test(line)), lines.join('\n'));
|
||||
});
|
||||
|
||||
test("a target follows the leader's repack by its id while it runs, even once newer builds push it off the list", async t => {
|
||||
const message = await leaderMessage(t);
|
||||
const eas = easBuilds([{ id: 'leader-repack', status: 'IN_QUEUE', platform: 'IOS', message }]);
|
||||
|
||||
@@ -129,13 +129,17 @@ test('the staged digest changes with any file, mode or link, and the share key w
|
||||
await rm(join(root, 'entry.js')); await symlink('package.json', join(root, 'entry.js')); digests.push(await stagedProjectDigest(root));
|
||||
await mkdir(join(root, 'empty')); digests.push(await stagedProjectDigest(root));
|
||||
assert.equal(new Set(digests).size, digests.length);
|
||||
const input = { scope: 'acme/example#39506405687.1', platform: 'ios', nativeBuildId: 'a81d5d80', easProfile: 'explore', environment: 'preview', environmentVariables: 'a'.repeat(64), project: digests[0] };
|
||||
const input = { scope: 'acme/example#39506405687.1', platform: 'ios', fingerprint: 'ea9bf84e1a5b0900000ba7c1e6b1c7e272e1306f', easProfile: 'explore', environment: 'preview', environmentVariables: 'a'.repeat(64), project: digests[0] };
|
||||
const key = hostedRepackShareKey(input);
|
||||
assert.match(key, /^[0-9a-f]{32}$/);
|
||||
for (const [name, other] of Object.entries({ scope: 'acme/example#39506405687.2', platform: 'android', nativeBuildId: 'b0000000', easProfile: 'other', environment: 'production',
|
||||
for (const [name, other] of Object.entries({ scope: 'acme/example#39506405687.2', platform: 'android', fingerprint: 'fc2446ef4e6d0000000000000000000000000000', easProfile: 'other', environment: 'production',
|
||||
environmentVariables: 'b'.repeat(64), project: digests[1] })) {
|
||||
assert.notEqual(hostedRepackShareKey({ ...input, [name]: other }), key, name);
|
||||
}
|
||||
// Which build a job found to repack does not matter, only its native code: a target that starts after the others'
|
||||
// repack finished finds that repack as its newest build of the fingerprint (brentvatne/euxy#179), and must still
|
||||
// join it rather than make another.
|
||||
assert.equal(hostedRepackShareKey({ ...input, nativeBuildId: '7f6b592a' }), hostedRepackShareKey({ ...input, nativeBuildId: '2ebcfd8d' }));
|
||||
});
|
||||
|
||||
test("only a build whose message ends with this key, for this platform, is another job's repack", () => {
|
||||
|
||||
Reference in new issue
Block a user