Diffing `target-graph`s was difficult because the locales kept shuffling.
This patch will keep the locales in alphabetical order.
MozReview-Commit-ID: GvGYF7j9ftq
--HG--
extra : rebase_source : 6a9aef0efd61c4f1aa7df48ca513311da203ccdb
Stop hardcoding `android`, since we want to use this for desktop too.
We could potentially remove the `android` platform from the bumper configs
at this point.
We strip `-nightly` from the `build_platform` before comparing against the
`l10n-changesets.json` platform list. If we want to support different sets
of ci and nightly locales, we could either:
- point at a second changesets file for ci. This could either be a flatfile
or json; either works. or,
- explicitly name `win32-nightly` etc. in the platform list, and stop removing
the `-nightly` from the `build_platform` before comparing. This means some
locales may have up to 10 different platforms listed. This may get unwieldy,
but would be explicit.
MozReview-Commit-ID: Fvpby92cXdg
--HG--
extra : rebase_source : 503ce9bd455d9845d6598ce2e06c4a355e737053
Add an optional 'arguments' key to the yaml description for toolchain
tasks. This is a list of strings to be passed to the script invocation.
This lets us set behaviour, e.g. selecting the version to build or
selecting targets from the task description instead of having to
hard-code those things in the build script itself. Where the same
script otherwise works for multiple configurations, that is easier
to update and simplifies supporting variants.
MozReview-Commit-ID: 30oJYnQaZ7A
--HG--
extra : rebase_source : bdd7bdc5f874d1329ff52d900cd1ac93df23c6dc
Run scripts with a `.py` filename extension through `./mach python`
so the normal enviroment with in-tree python libraries is available.
This is helpful for the Rust upstream release download and repackaging
steps, which are more easily expressed in python than in an sh-based
build script like we use for clang and other tools.
Invocation of `mach python` on Windows-hosted generic workers fails
because of some missing environment pieces. For the purposes of this
bug we can just run the script for Windows targets in a docker-worker
so Python support was left unimplemented for generic workers.
MozReview-Commit-ID: 4XUQ1XrVBc2
--HG--
extra : rebase_source : df81d2da7e70fffb29e96377f16ab22def9e94e0
This adds a download of allthethings.json to the Decision Task (in optimize phase).
If there is a network error, in order to avoid a broken decision task, we treat all buildbot-bridge jobs as valid.
MozReview-Commit-ID: GpKVV5pUwGL
--HG--
extra : rebase_source : 39e853f98dcfafe252504f724cdf84b95b0eece8
Strictly speaking we don't need all files in the directories listed
in the profile. But the checkout is still small enough and it is far
less effort than cherry-picking every file needed by every toolchain
task.
This brings the checkout down to ~3700 files, which only takes 1-2s.
MozReview-Commit-ID: 2BpKdZ2Pvx9
--HG--
extra : rebase_source : e9589b29f7e1b60ac7ee4e8050689dc3d5a8f418
extra : source : 2c0d6e566cc719823515ec2403e1d6b2ace955aa
This continues to use a file-based reader when run locally.
MozReview-Commit-ID: CJuYKDj2E3n
--HG--
extra : rebase_source : ceb56f3eb5b56fca90f19736ab710696fde86bd1
This means that a push to try affecting only Android will only run android builds
and tests, for example.
MozReview-Commit-ID: HVUvIg0EUZn
--HG--
extra : rebase_source : 8b080b7648cea5f82cd03c9a7950667277b75118
extra : source : b41cd667697e13c989659b16bf649090a3908ecd
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : f208e4960cf76a9dfe77634b87f0058f676e9fa9
extra : source : 046d705929f7a41e977eec19c8503afccdec7592
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : 25e9966696d78d899783d9f38533d5ae66f9ccb9
extra : source : b53ff084c2d7968a1d9864d1343f2d9381fb652b
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
--HG--
extra : rebase_source : cf1654036041a64398a2cd38e35e8de4f3596ff9
extra : source : c25af2c111a5be4e3381d0b002641691d15fe4e8
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
--HG--
extra : rebase_source : ba0aa536e8c474b36c63d1447c83ed9885f1e3e6
extra : source : a3b7bdfdb116300daa3f49e0dfc96177e1369440
Strictly speaking we don't need all files in the directories listed
in the profile. But the checkout is still small enough and it is far
less effort than cherry-picking every file needed by every toolchain
task.
This brings the checkout down to ~3700 files, which only takes 1-2s.
MozReview-Commit-ID: 2BpKdZ2Pvx9
--HG--
extra : rebase_source : 916c6722c6d495cdccea0076c673d75e4ee48f28
extra : source : 2c0d6e566cc719823515ec2403e1d6b2ace955aa
This continues to use a file-based reader when run locally.
MozReview-Commit-ID: CJuYKDj2E3n
--HG--
extra : rebase_source : 8532c2cd8bff035b6b7f497947356383713944d7
This means that a push to try affecting only Android will only run android builds
and tests, for example.
MozReview-Commit-ID: HVUvIg0EUZn
--HG--
extra : rebase_source : 80a41a86ba892ccfe56bcbf6b4daf03a60424a5f
extra : source : b41cd667697e13c989659b16bf649090a3908ecd
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : 4cf3efc9c57bb14d2f44147c8881d0a0a18703d6
extra : source : 046d705929f7a41e977eec19c8503afccdec7592
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : 03a10610aa3337269fe76a1196bb9b1665e1ab20
extra : source : b53ff084c2d7968a1d9864d1343f2d9381fb652b
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
--HG--
extra : rebase_source : c6e2ff90316d43cd93826de5c30a1936f19c01ca
extra : source : c25af2c111a5be4e3381d0b002641691d15fe4e8
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
--HG--
extra : rebase_source : d5bce42fc0ea24616d885eed62e5e5a42b4fce24
extra : source : a3b7bdfdb116300daa3f49e0dfc96177e1369440
The previous patch was ineffective, because the noted transform over-wrote
the yml configuration.
MozReview-Commit-ID: ICENT0DGzxy
--HG--
extra : source : e024769dac36e22935f1e64a328bb15334d6bdef
