also clean up and move more config to the promotion config.
MozReview-Commit-ID: FmTWNNPcEaZ
--HG--
extra : rebase_source : 40431217fafb6796dbd65c7dfeab0e891ac1bbd4
extra : source : 0f5418a83477c1b6b221e4d28515792410e504d0
There are e.g. some build infrastructure changes that we want to have a
controlled impact on the Firefox builds we produce. We have, in multiple
occasions, gone through manual work to compare Firefox builds, most of
the time using the diffoscope tool (https://diffoscope.org/).
This change introduces a new task kind that takes two Firefox builds as
input, either by name (reference to a build from the current task graph)
or by index (reference to a build from a previous push), and compares
them.
In order to get a Firefox build by index, we rely on dummy tasks with
an optimization we expect to always hit, so we add the necessary bits
to ensure those dummy tasks can go through up to the optimization phase
and be optimized out there.
--HG--
extra : rebase_source : 37482f67652dab2fcef2db4e6b8efe653999bae5
Add support for the three firefox and devedition relpro flavors (we
could probably reduce devedition to 2).
Also, instead of defining which kinds to use from the previous graph
in `previous_graph_kinds`, specify which kinds to rebuild (ignore)
from the previous graph in `rebuild_kinds`. This list will be much
smaller (currently empty).
MozReview-Commit-ID: 5rH1TW7GbAD
--HG--
extra : rebase_source : b4294a0d17a99b2ffd48f5d62821c724324b242c
extra : histedit_source : b3fe87101e4595f5fc5c7daaa2d4a0bd88418667
The runnable jobs API on Treeherder is timing out quite frequently.
This is due that it downloads the full-task-graph.json artifact from the Gecko decision task.
This file has grown to be massive (over 30MB) and that takes too long to fetch. The API times out after 20 seconds.
The data we need from the artifact is minimal. We can generate a file with just the minimum amount of data needed.
This code change adds logic to generate a runnable-jobs.json.gz file after the full-task-graph.json artifact is generated.
MozReview-Commit-ID: 9u2H7HbUAcl
--HG--
extra : rebase_source : f450527e7dd565e29cfe2e04dc52761e77827cc0
This allows firefox and thunderbird builds to avoid using each others bits.
MozReview-Commit-ID: KYQYDd2tkGj
--HG--
extra : rebase_source : 42f1d13ec609f066cb3bd3050ed894296b72d982
These two new attributes will help us determine which tasks belong in which release promotion graphs. In the future, we'll specify these for all shippable builds as well, and we can reduce the usage of the `product` keys. However, sometimes we need `product` to differ from `shipping-product` (e.g., `mobile` vs `fennec`; in this case we mean `stage_product` vs `shipping_product`), so I haven't yet touched those key/value pairs.
MozReview-Commit-ID: LEuf4CS277Q
--HG--
extra : rebase_source : 4971883500658ace209bec97a639a54c94292b50
Instead of relying on environment variables, pass these in as parameters.
MozReview-Commit-ID: An58Bu2kd1g
--HG--
extra : rebase_source : 9c7b6beb13b676a3376a897f2c8143cc042b8276
This template takes a single integer as input, and sets the 'task_duplicates'
attribute on any tasks that were explicitly specified by try_task_config.json.
This means dependent tasks or 'always_target' tasks will not be rebuilt.
To support this template, the apply_jsone_templates morph now requires the
entire try_task_config object instead of just the templates.
MozReview-Commit-ID: DwxUtlC5VD5
--HG--
extra : rebase_source : c57a80ada2fc4591a5ce4c94965b8c9f02bb53b1
The motivation for this change is to create a 'rebuild' template that sets
'attributes.task_duplicates'. A nice side effect is that if any of 'task'
or 'attributes' aren't explicitly set, we can ignore the result which means
we don't need to have a whole bunch of 'else' clauses in the templates.
MozReview-Commit-ID: DX3UyFGb4ff
--HG--
extra : rebase_source : b768cb9f9df3cb22827209519f1cd2a4944af74f
Tasks that have the 'always_target' attribute set will be always be included
in the target_task_graph, regardless of target task filtering.
Furthermore, if they were only added because of this attribute (i.e, the
filters would have excluded the task), then the task will be a candidate for
optimization even if the 'optimize_target_tasks' parameter is False.
MozReview-Commit-ID: 9eoVJ5qpAMO
--HG--
extra : rebase_source : 9635002720d088ca9870649f3143d6293c666610
This patch adds the `release-notify-promote` and `release-notify-publish` kinds. It also genericizes all the notifications, and updates the kinds that use those notifications.
MozReview-Commit-ID: 9ymXKzthVF4
--HG--
extra : rebase_source : 6d7db48afa14e4124834321784da65e8d1e84c98
This patch adds the beetmover-cdns kind, and adds it to
`publish_fennec`.
This was the first non-buildbot-bridge, non-dummy relpro task, so this
needed a new transform.
This patch also updates the `previous_graph_kinds` and updates the
beetmover scopes in scriptworker.py.
MozReview-Commit-ID: 3rpkjuLjjXz
--HG--
extra : rebase_source : d491f1ce5d10ee8f2602698236106007e203c358
This patch adds the initial `release-version-bump` kind, and adds it to
the `publish_fennec` `target_tasks_method`.
It also adds support for `next_version`.
MozReview-Commit-ID: 9YRswddeuZ3
--HG--
extra : rebase_source : 991cbf89b40c4ef980c42251001e81be5cedaf00
This patch adds the initial `release-uptake-monitoring` kind, and adds
it to the `publish_fennec` `target_tasks_method`.
MozReview-Commit-ID: 3RDMNGrbBwD
--HG--
extra : rebase_source : f504c8e173337d45bc9f374bd0349c6823b50ffb
This patch adds the initial `release-bouncer-aliases` kind, and adds it
to the `publish_fennec` `target_tasks_method`.
It also adds the ability to specify the `tuxedo_server_url`
`by-project`.
MozReview-Commit-ID: 9I4IaUlbCCD
--HG--
extra : rebase_source : d0ed88a687ef6fb9f97dc67c0f04742bbfeb201d
This patch adds the initial `release-mark-as-shipped` kind, and adds it to
the `publish_fennec` `target_tasks_method`.
MozReview-Commit-ID: F8AYscJQWlh
--HG--
extra : rebase_source : 9522b95f65b5b373a54bc0fc01a20c72adbee0cb
This patch adds the first releasetask as a new kind. To support this, we
added a new `release-promotion` flag in the buildbot job. If this is
set, we use the new `bb_release_worker` function; otherwise we fall back
to the `bb_ci_worker` function (this is the old behavior, factored out
into a separate function).
We also added `build_number` and `release_promotion` attributes in the
task definition. Finally, `build_number` now defaults to 1, allowing us
to create the task graph locally without forcing us to set
`BUILD_NUMBER` in the environment.
MozReview-Commit-ID: 8vNMHJemqAG
--HG--
extra : rebase_source : c8816d3e7b2f5358ea51ee8d5fb12585e67e9853
Add a release promotion custom action for releng's TC release promotion migration work.
This action generates a graph dependent on previously built tasks. To track these, we add the `do_not_optimize` and `existing_tasks` parameters. The `do_not_optimize` parameter specifies tasks that we want to explicitly exclude from taskgraph optimization. The `existing_tasks` parameter specifies a label-to-taskid map for tasks from previous graphs.
MozReview-Commit-ID: 1vKrNUavM4V
--HG--
extra : rebase_source : b8ba95d270aafe1464c2b3bfc318b9568500a7a1
With this in place, all `-j`obs will not run by default on try. This will omit
such jobs in most try pushes even if files-changed matches. This is
unfortunate, but better than running them unconditionally. Fuzzy selections,
and later `just try it` pushes, are the ultimate solution here.
With this change, a push with no try syntax or try_task_config.json will schedule
no tasks at all.
MozReview-Commit-ID: FGjqlDW1FT6
--HG--
extra : rebase_source : 727ceafb1b6d24f83c0c7382b6a877ecb65863ab
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : 25e9966696d78d899783d9f38533d5ae66f9ccb9
extra : source : b53ff084c2d7968a1d9864d1343f2d9381fb652b
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
--HG--
extra : rebase_source : cf1654036041a64398a2cd38e35e8de4f3596ff9
extra : source : c25af2c111a5be4e3381d0b002641691d15fe4e8
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : 03a10610aa3337269fe76a1196bb9b1665e1ab20
extra : source : b53ff084c2d7968a1d9864d1343f2d9381fb652b
