We want to be able to retroactively tell whether a push was a backstop or not.
This patch stores whether or not a push was a "backstop" directly in the
parameters. The optimization strategy now simply returns 'not
params["backstop"]'.
For simplicity, I'm not counting the 'optimized-backstop' as a backstop. It's
unclear if we'll want to be able to detect these types of the pushes in the
future or not, but we can cross that bridge when we get there.
Differential Revision: https://phabricator.services.mozilla.com/D88151
In the past, the 'backstop' optimization was applied to tasks by default across
all projects, even though it only really made sense on autoland. To choose what
would happen on non-autoland branches, we invented this 'remove_on_projects'
concept.
These days, we only apply the backstop optimization in the first place for
autoland. So 'remove_on_projects' is no longer necessary.
Depends on D88149
Differential Revision: https://phabricator.services.mozilla.com/D88150
This patch cleans up some of the backstop strategy names. Specifically:
1. Rename 'full-backstop' -> 'backstop'. The old 'backstop' algorithm was
unused anyway, so there is no conflict. It is also just defined directly in
the decorator rather than using 'Alias'.
So now rather than 'full-backstop' and 'optimized-backstop', it's just
'backstop' and 'optimized-backstop'.
2. Remove 'backstop-X-hours-Y-minutes' strategies, and replace them with
the corresponding 'push-interval-X' strategy.
This means we lose the time component in the 'optimized-backstop'. But it isn't
a problem, because we shouldn't be using a time component there at all anyway
(we should just use it with the 'backstop').
Differential Revision: https://phabricator.services.mozilla.com/D88149
Using nested Any generates deeply nested non-trivial schemas because we
also end up nesting optionally_keyed_by's, and the addition of one field
in one optionally_keyed_by in bug 1657769 made the whole schema
compilation step an order of magnitude slower.
Even before bug 1657769, the schema compilation step was slow for tests,
and with this change, it's virtually instantaneous.
Differential Revision: https://phabricator.services.mozilla.com/D88543
Certain tasks are meant to perform follow-up work to another task. For example,
a 'signing' task signs an artifact from a dependency task. An 'upload-symbols'
tasks uploads an artifact from a dependency task.
In general, we only want to run these types of tasks when the dependency task
is running. But in practice, these tasks often cause the dependency to get
pulled in.
We've tried to fix this in the past by making the 'run-on-projects' and
'optimization' keys match their primary dependency. But it's very easy to
mess this up, and some optimizations (like bugbug) take the content of the
task definition into account. So even if both tasks use the same optimization,
they could have different results.
This revision adds a new 'if-dependencies' key that denotes a task that should
only run if one of the dependencies specified in this list is also run. It
will allow us to stop playing whack-a-mole when trying to make sure tasks
aren't being pulled in solely due to these types of dependencies.
This feature is implemented independently of the 'optimization', so tasks that
specify this can still be optimized even if their primary dependency does run.
Differential Revision: https://phabricator.services.mozilla.com/D80508
Edges are defined as a (left, right, name) tuple, which means task left depends on task right using an edge called name. Previously the optimization code never took dependencies into account, so the mock graph generator never bothered to create them. But in the child revision, we start using these values, so the generation should set it up appropriately.
I had to make it conditional due to the way the assertions in the test_get_subgraph test are set up. I figured it was easier to do that than to re-write the assertions.
Depends on D88483
Differential Revision: https://phabricator.services.mozilla.com/D88484
This changes most of our automation builds to clang 11.0.0 rc2.
Not included:
* code coverage builds, per bug 1660341
* mingw builds, which have traditionally been on their own update cadence, and in this case are blocked anyway by bug 1658632
This will leave some unused clang-9 task definitions. I intend to clean them up, but at a later date. For now I want to focus on making sure this update sticks, since patches like this have a tendency to bounce.
Differential Revision: https://phabricator.services.mozilla.com/D88313
This is a fork of the release-partner-repack-beetmover kind and transform. It's modified to cope with having one upstream task with many partner builds, rather than a many beetmover tasks dealing with a single config-platform-locale combination.
Differential Revision: https://phabricator.services.mozilla.com/D87730
A single task is created to do all partner attributions. The partner_attribution transform processes the configuration into an environment variable for the tools/attribution/attribute.py script to use. This is quite verbose so a large number of configurations may cause problems.
Applies the same priority modification to attribution tasks as to partner repacks, to not impede the main part of the graph.
Differential Revision: https://phabricator.services.mozilla.com/D87729
The partner attribution config is stored in the same repository as the repo manifest for partner repacks, but all in attribution_config.yml instead of default.xml. This extends the existing support for using the Github API to read files to retrieve and process the attribution config.
Differential Revision: https://phabricator.services.mozilla.com/D87728
Renames the release_enable_partners parameter to release_enable_partner_repack, and adds release_enable_partner_attribution for attribution. This it to provide support for disabling them independently in main releases, and in respins.
Adds docs for attribution, update docs for repacks.
Hardwire values for the enable params for the respin flavors, other wise read from the input (defaulting to on in promotion, off otherwise).
Fixes up the rebuild-kinds for partner repacks so that they reflect the current set, although the top level may be all that is needed.
Differential Revision: https://phabricator.services.mozilla.com/D87727
This improves the integrity of downloads of upstream artifacts when using fetch-content. If `verify-hash: True` is set on the fetch config, then the chain-of-trust.json of the upstream is used to retieve the expected sha256 of the artifact, and this is checked.
Differential Revision: https://phabricator.services.mozilla.com/D87725
Prior to this patch the task graph would always include a release-partner-repack-<platform> task for all 6 platforms, regardless of what was specified in release_partner_config. This was particularly obvious in the off-cycle respin scenario when a single partner is repacked. By moving and reusing get_repack_ids_by_platform() it's easy to skip unneeded platforms.
Differential Revision: https://phabricator.services.mozilla.com/D87724
This is a fork of the release-partner-repack-beetmover kind and transform. It's modified to cope with having one upstream task with many partner builds, rather than a many beetmover tasks dealing with a single config-platform-locale combination.
Differential Revision: https://phabricator.services.mozilla.com/D87730
A single task is created to do all partner attributions. The partner_attribution transform processes the configuration into an environment variable for the tools/attribution/attribute.py script to use. This is quite verbose so a large number of configurations may cause problems.
Applies the same priority modification to attribution tasks as to partner repacks, to not impede the main part of the graph.
Differential Revision: https://phabricator.services.mozilla.com/D87729
The partner attribution config is stored in the same repository as the repo manifest for partner repacks, but all in attribution_config.yml instead of default.xml. This extends the existing support for using the Github API to read files to retrieve and process the attribution config.
Differential Revision: https://phabricator.services.mozilla.com/D87728
Renames the release_enable_partners parameter to release_enable_partner_repack, and adds release_enable_partner_attribution for attribution. This it to provide support for disabling them independently in main releases, and in respins.
Adds docs for attribution, update docs for repacks.
Hardwire values for the enable params for the respin flavors, other wise read from the input (defaulting to on in promotion, off otherwise).
Fixes up the rebuild-kinds for partner repacks so that they reflect the current set, although the top level may be all that is needed.
Differential Revision: https://phabricator.services.mozilla.com/D87727
This improves the integrity of downloads of upstream artifacts when using fetch-content. If `verify-hash: True` is set on the fetch config, then the chain-of-trust.json of the upstream is used to retieve the expected sha256 of the artifact, and this is checked.
Differential Revision: https://phabricator.services.mozilla.com/D87725
Prior to this patch the task graph would always include a release-partner-repack-<platform> task for all 6 platforms, regardless of what was specified in release_partner_config. This was particularly obvious in the off-cycle respin scenario when a single partner is repacked. By moving and reusing get_repack_ids_by_platform() it's easy to skip unneeded platforms.
Differential Revision: https://phabricator.services.mozilla.com/D87724
This changes most of our automation builds to clang 11.0.0 rc2.
Not included:
* code coverage builds, per bug 1660341
* mingw builds, which have traditionally been on their own update cadence, and in this case are blocked anyway by bug 1658632
This will leave some unused clang-9 task definitions. I intend to clean them up, but at a later date. For now I want to focus on making sure this update sticks, since patches like this have a tendency to bounce.
Differential Revision: https://phabricator.services.mozilla.com/D88313
This patch fixes an issue in the custom-v8 toolchain where there is tooling missing that is needed to build v8 (the libtinfo5 package).
Differential Revision: https://phabricator.services.mozilla.com/D88310
From mwoodrow:
bcv is 'basic compositor video', which explicitly disables acceleration using a
pref and then runs some video tests. For normal -qr platforms this isn't
useful, since the pref would disable WebRender, and it'd be an identical
configuation to the non-qr variant.
When software webrender is enabled, disabling acceleration won't disable WR,
and test remains useful (testing video performance of the software webrender
configuration).
The baseline I want to compare against here is the non-qr bcv, to see if
webrender-software is worse than layers-software.
Differential Revision: https://phabricator.services.mozilla.com/D88192
It's often useful to nest 'by-*' statements in the CI configs. However, calling
'resolve_keyed_by' will keep drilling down and resolving these statements all
at once. Normally this is fine, but sometimes we may want to use, e.g 'by-foo'
when 'foo' hasn't been resolved yet (i.e, it is being set by a later transform
or something like that).
Currently, this means that we'll hit the 'default' case as 'foo' doesn't exist.
This revision adds the ability to 'defer' evaluation of 'foo' until a later
time.
While running into this scenario might be a code smell, re-organizing
transforms to untangle these kinds of ordering dependencies will be a major
undertaking. Plus, there are legitimate reasons we might want one kind of
ordering with certain tasks, then another ordering for other tasks.
Differential Revision: https://phabricator.services.mozilla.com/D88191
This adds toolchain definitions for clang 11.0.0 rc2, so that developers can get a sneak peek, but nothing in automation uses these tasks yet. We'll make the switch in a later patch.
NB: most of `clang.yml` is rote copy-paste, except for `macosx64-clang-11` which makes a deliberate departure, described in a comment.
Differential Revision: https://phabricator.services.mozilla.com/D88189
Also reflects the move from CraneStation to the WebAssembly account. And we need a small tweak to the build script to accommodate one of the changes that got picked up along the way.
Differential Revision: https://phabricator.services.mozilla.com/D88187
This avoids a set of intermittent issues related to `zstd` decompression failures, which in the absence of these changes break the entire build.
This also requires [updating an environment variable](https://github.com/mozilla/sccache/pull/822), which we do in `client.mk` as well as documentation.
Differential Revision: https://phabricator.services.mozilla.com/D88184
In Bug 1466660, we started deleting the fetches after a task had run, to avoid
interference between tasks. It turns out the only tasks this was for were the
`source-test-jsshell` tasks, which were changed to use an absolute directory in
Bug 1465181. However, since Bug 1568460 we've always used a per-task directory
for fetches, so can remove the work-around of removing fethes.
Differential Revision: https://phabricator.services.mozilla.com/D86670
We define a value `run.workdir` to point to the defaut directory to use for
all sorts of paths, including checkouts, in the job transform. However, that
directory only makes sense for docker-worker tasks, and relative paths should
be used elsewhere. Adjust the paths on non-docker-worker linux tasks to match
macOS paths.
Differential Revision: https://phabricator.services.mozilla.com/D86668
