Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
# This Source Code Form is subject to the terms of the Mozilla Public
|
|
|
|
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
|
|
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
|
|
"""
|
|
|
|
These transformations take a task description and turn it into a TaskCluster
|
|
|
|
task definition (along with attributes, label, etc.). The input to these
|
|
|
|
transformations is generic to any kind of task, but abstracts away some of the
|
|
|
|
complexities of worker implementations, scopes, and treeherder annotations.
|
|
|
|
"""
|
|
|
|
|
2016-08-31 15:24:54 +00:00
|
|
|
from __future__ import absolute_import, print_function, unicode_literals
|
|
|
|
|
2017-09-06 23:09:15 +00:00
|
|
|
import hashlib
|
2017-04-17 00:02:38 +00:00
|
|
|
import os
|
2017-08-18 22:30:32 +00:00
|
|
|
import re
|
2016-09-12 15:51:49 +00:00
|
|
|
import time
|
2017-04-06 16:50:41 +00:00
|
|
|
from copy import deepcopy
|
2020-02-07 18:12:23 +00:00
|
|
|
import six
|
2019-07-12 20:44:12 +00:00
|
|
|
from six import text_type
|
2016-09-12 15:51:49 +00:00
|
|
|
|
2019-07-02 17:05:21 +00:00
|
|
|
import attr
|
|
|
|
|
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-18 21:07:03 +00:00
|
|
|
from mozbuild.util import memoize
|
2017-07-20 01:24:12 +00:00
|
|
|
from taskgraph.util.attributes import TRUNK_PROJECTS
|
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-18 21:07:03 +00:00
|
|
|
from taskgraph.util.hash import hash_path
|
2019-07-26 21:55:27 +00:00
|
|
|
from taskgraph.util.taskcluster import get_root_url
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
from taskgraph.util.treeherder import split_symbol
|
2017-02-01 00:30:52 +00:00
|
|
|
from taskgraph.transforms.base import TransformSequence
|
2019-03-07 04:20:36 +00:00
|
|
|
from taskgraph.util.keyed_by import evaluate_keyed_by
|
2018-02-20 18:26:32 +00:00
|
|
|
from taskgraph.util.schema import (
|
|
|
|
validate_schema,
|
|
|
|
Schema,
|
|
|
|
optionally_keyed_by,
|
|
|
|
resolve_keyed_by,
|
|
|
|
OptimizationSchema,
|
2018-09-25 20:26:55 +00:00
|
|
|
taskref_or_string,
|
2018-02-20 18:26:32 +00:00
|
|
|
)
|
2019-06-21 22:12:59 +00:00
|
|
|
from taskgraph.util.partners import get_partners_to_be_published
|
2018-02-16 02:14:26 +00:00
|
|
|
from taskgraph.util.scriptworker import (
|
|
|
|
BALROG_ACTIONS,
|
|
|
|
get_release_config,
|
|
|
|
)
|
2018-08-20 18:29:58 +00:00
|
|
|
from taskgraph.util.signed_artifacts import get_signed_artifacts
|
2019-07-12 20:44:12 +00:00
|
|
|
from taskgraph.util.workertypes import worker_type_implementation
|
2019-03-01 19:44:29 +00:00
|
|
|
from voluptuous import Any, Required, Optional, Extra, Match
|
2017-12-16 21:09:25 +00:00
|
|
|
from taskgraph import GECKO, MAX_DEPENDENCIES
|
2017-08-23 15:53:56 +00:00
|
|
|
from ..util import docker as dockerutil
|
2019-03-19 19:13:56 +00:00
|
|
|
from ..util.workertypes import get_worker_type
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2018-05-05 00:23:31 +00:00
|
|
|
RUN_TASK = os.path.join(GECKO, 'taskcluster', 'scripts', 'run-task')
|
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-18 21:07:03 +00:00
|
|
|
|
2019-09-09 16:51:42 +00:00
|
|
|
SCCACHE_GCS_PROJECT = 'sccache-3'
|
|
|
|
|
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-18 21:07:03 +00:00
|
|
|
|
|
|
|
@memoize
|
|
|
|
def _run_task_suffix():
|
|
|
|
"""String to append to cache names under control of run-task."""
|
|
|
|
return hash_path(RUN_TASK)[0:20]
|
|
|
|
|
|
|
|
|
2019-04-16 09:46:07 +00:00
|
|
|
def _compute_geckoview_version(app_version, moz_build_date):
|
|
|
|
"""Geckoview version string that matches geckoview gradle configuration"""
|
|
|
|
# Must be synchronized with /mobile/android/geckoview/build.gradle computeVersionCode(...)
|
|
|
|
version_without_milestone = re.sub(r'a[0-9]', '', app_version, 1)
|
|
|
|
parts = version_without_milestone.split('.')
|
|
|
|
return "%s.%s.%s" % (parts[0], parts[1], moz_build_date)
|
|
|
|
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
# A task description is a general description of a TaskCluster task
|
|
|
|
task_description_schema = Schema({
|
|
|
|
# the label for this task
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('label'): text_type,
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
|
|
|
# description of the task (for metadata)
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('description'): text_type,
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
|
|
|
# attributes for this task
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('attributes'): {text_type: object},
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2017-08-31 20:38:08 +00:00
|
|
|
# relative path (from config.path) to the file task was defined in
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('job-from'): text_type,
|
2017-08-31 20:38:08 +00:00
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
# dependencies of this task, keyed by name; these are passed through
|
|
|
|
# verbatim and subject to the interpretation of the Task's get_dependencies
|
|
|
|
# method.
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('dependencies'): {text_type: object},
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2019-03-04 17:07:34 +00:00
|
|
|
# Soft dependencies of this task, as a list of tasks labels
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('soft-dependencies'): [text_type],
|
2019-03-04 17:07:34 +00:00
|
|
|
|
2018-01-08 15:14:26 +00:00
|
|
|
Optional('requires'): Any('all-completed', 'all-resolved'),
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
# expiration and deadline times, relative to task creation, with units
|
2016-08-31 15:24:54 +00:00
|
|
|
# (e.g., "14 days"). Defaults are set based on the project.
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('expires-after'): text_type,
|
|
|
|
Optional('deadline-after'): text_type,
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
|
|
|
# custom routes for this task; the default treeherder routes will be added
|
|
|
|
# automatically
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('routes'): [text_type],
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
|
|
|
# custom scopes for this task; any scopes required for the worker will be
|
2017-08-08 19:13:05 +00:00
|
|
|
# added automatically. The following parameters will be substituted in each
|
|
|
|
# scope:
|
|
|
|
# {level} -- the scm level of this push
|
2017-12-06 03:35:45 +00:00
|
|
|
# {project} -- the project of this push
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('scopes'): [text_type],
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2017-01-25 21:04:37 +00:00
|
|
|
# Tags
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('tags'): {text_type: text_type},
|
2017-01-25 21:04:37 +00:00
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
# custom "task.extra" content
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('extra'): {text_type: object},
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
|
|
|
# treeherder-related information; see
|
2020-01-24 15:52:50 +00:00
|
|
|
# https://firefox-ci-tc.services.mozilla.com/schemas/taskcluster-treeherder/v1/task-treeherder-config.json
|
2016-08-31 15:24:54 +00:00
|
|
|
# If not specified, no treeherder extra information or routes will be
|
|
|
|
# added to the task
|
|
|
|
Optional('treeherder'): {
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
# either a bare symbol, or "grp(sym)".
|
2020-01-21 17:12:08 +00:00
|
|
|
'symbol': text_type,
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
|
|
|
# the job kind
|
|
|
|
'kind': Any('build', 'test', 'other'),
|
|
|
|
|
|
|
|
# tier for this task
|
|
|
|
'tier': int,
|
|
|
|
|
|
|
|
# task platform, in the form platform/collection, used to set
|
|
|
|
# treeherder.machine.platform and treeherder.collection or
|
|
|
|
# treeherder.labels
|
2019-03-01 19:44:29 +00:00
|
|
|
'platform': Match('^[A-Za-z0-9_-]{1,50}/[A-Za-z0-9_-]{1,50}$'),
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
},
|
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
# information for indexing this build so its artifacts can be discovered;
|
|
|
|
# if omitted, the build will not be indexed.
|
|
|
|
Optional('index'): {
|
|
|
|
# the name of the product this build produces
|
2020-01-21 17:12:08 +00:00
|
|
|
'product': text_type,
|
2016-09-12 15:51:49 +00:00
|
|
|
|
|
|
|
# the names to use for this job in the TaskCluster index
|
2020-01-21 17:12:08 +00:00
|
|
|
'job-name': text_type,
|
2016-09-28 11:20:41 +00:00
|
|
|
|
2017-01-09 21:23:04 +00:00
|
|
|
# Type of gecko v2 index to use
|
2017-01-12 16:17:32 +00:00
|
|
|
'type': Any('generic', 'nightly', 'l10n', 'nightly-with-multi-l10n',
|
2019-05-28 12:42:19 +00:00
|
|
|
'nightly-l10n', 'shippable', 'shippable-l10n',
|
2019-04-16 09:46:07 +00:00
|
|
|
'android-nightly', 'android-nightly-with-multi-l10n'),
|
2017-01-09 21:23:04 +00:00
|
|
|
|
2016-09-28 11:20:41 +00:00
|
|
|
# The rank that the task will receive in the TaskCluster
|
|
|
|
# index. A newly completed task supercedes the currently
|
|
|
|
# indexed task iff it has a higher rank. If unspecified,
|
|
|
|
# 'by-tier' behavior will be used.
|
|
|
|
'rank': Any(
|
2016-11-03 02:23:27 +00:00
|
|
|
# Rank is equal the timestamp of the build_date for tier-1
|
2016-09-28 11:20:41 +00:00
|
|
|
# tasks, and zero for non-tier-1. This sorts tier-{2,3}
|
|
|
|
# builds below tier-1 in the index.
|
|
|
|
'by-tier',
|
|
|
|
|
|
|
|
# Rank is given as an integer constant (e.g. zero to make
|
|
|
|
# sure a task is last in the index).
|
|
|
|
int,
|
|
|
|
|
2016-11-03 02:23:27 +00:00
|
|
|
# Rank is equal to the timestamp of the build_date. This
|
2016-09-28 11:20:41 +00:00
|
|
|
# option can be used to override the 'by-tier' behavior
|
|
|
|
# for non-tier-1 tasks.
|
2016-11-03 02:23:27 +00:00
|
|
|
'build_date',
|
2016-09-28 11:20:41 +00:00
|
|
|
),
|
2016-09-12 15:51:49 +00:00
|
|
|
},
|
|
|
|
|
|
|
|
# The `run_on_projects` attribute, defaulting to "all". This dictates the
|
|
|
|
# projects on which this task should be included in the target task set.
|
|
|
|
# See the attributes documentation for details.
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('run-on-projects'): optionally_keyed_by('build-platform', [text_type]),
|
2016-09-12 15:51:49 +00:00
|
|
|
|
2018-10-29 11:11:46 +00:00
|
|
|
# Like `run_on_projects`, `run-on-hg-branches` defaults to "all".
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('run-on-hg-branches'): optionally_keyed_by('project', [text_type]),
|
2018-10-29 11:11:46 +00:00
|
|
|
|
2017-11-09 02:02:28 +00:00
|
|
|
# The `shipping_phase` attribute, defaulting to None. This specifies the
|
|
|
|
# release promotion phase that this task belongs to.
|
2017-12-08 03:23:57 +00:00
|
|
|
Required('shipping-phase'): Any(
|
2017-11-09 02:02:28 +00:00
|
|
|
None,
|
2017-12-06 03:35:45 +00:00
|
|
|
'build',
|
2017-11-09 02:02:28 +00:00
|
|
|
'promote',
|
2017-12-06 03:35:45 +00:00
|
|
|
'push',
|
2017-11-09 02:02:28 +00:00
|
|
|
'ship',
|
|
|
|
),
|
|
|
|
|
|
|
|
# The `shipping_product` attribute, defaulting to None. This specifies the
|
|
|
|
# release promotion product that this task belongs to.
|
2017-12-08 03:23:57 +00:00
|
|
|
Required('shipping-product'): Any(
|
2017-11-09 02:02:28 +00:00
|
|
|
None,
|
2020-01-21 17:12:08 +00:00
|
|
|
text_type
|
2017-11-09 02:02:28 +00:00
|
|
|
),
|
|
|
|
|
2017-10-30 13:41:51 +00:00
|
|
|
# The `always-target` attribute will cause the task to be included in the
|
|
|
|
# target_task_graph regardless of filtering. Tasks included in this manner
|
|
|
|
# will be candidates for optimization even when `optimize_target_tasks` is
|
|
|
|
# False, unless the task was also explicitly chosen by the target_tasks
|
|
|
|
# method.
|
2017-12-08 03:23:57 +00:00
|
|
|
Required('always-target'): bool,
|
2017-10-30 13:41:51 +00:00
|
|
|
|
2017-08-01 20:02:59 +00:00
|
|
|
# Optimization to perform on this task during the optimization phase.
|
|
|
|
# Optimizations are defined in taskcluster/taskgraph/optimize.py.
|
2018-02-20 18:26:32 +00:00
|
|
|
Required('optimization'): OptimizationSchema,
|
2017-03-10 18:17:43 +00:00
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
# the provisioner-id/worker-type for the task. The following parameters will
|
|
|
|
# be substituted in this string:
|
|
|
|
# {level} -- the scm level of this push
|
2020-01-21 17:12:08 +00:00
|
|
|
'worker-type': text_type,
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2016-11-10 18:02:54 +00:00
|
|
|
# Whether the job should use sccache compiler caching.
|
2017-12-08 03:23:57 +00:00
|
|
|
Required('needs-sccache'): bool,
|
2016-11-10 18:02:54 +00:00
|
|
|
|
2018-08-20 18:29:15 +00:00
|
|
|
# Set of artifacts relevant to release tasks
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('release-artifacts'): [text_type],
|
2018-08-20 18:29:15 +00:00
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
# information specific to the worker implementation that will run this task
|
2019-07-02 17:05:21 +00:00
|
|
|
Optional('worker'): {
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('implementation'): text_type,
|
Bug 1359976: base worker payload generation on worker-type; r=wcosta r=aki
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
--HG--
extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
2017-05-08 22:53:50 +00:00
|
|
|
Extra: object,
|
2019-08-27 03:06:47 +00:00
|
|
|
},
|
|
|
|
|
|
|
|
# Override the default priority for the project
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('priority'): text_type,
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
})
|
|
|
|
|
2017-10-12 20:03:32 +00:00
|
|
|
TC_TREEHERDER_SCHEMA_URL = 'https://github.com/taskcluster/taskcluster-treeherder/' \
|
|
|
|
'blob/master/schemas/task-treeherder-config.yml'
|
|
|
|
|
2017-09-18 12:36:36 +00:00
|
|
|
|
2018-01-12 20:49:27 +00:00
|
|
|
UNKNOWN_GROUP_NAME = "Treeherder group {} (from {}) has no name; " \
|
|
|
|
"add it to taskcluster/ci/config.yml"
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
V2_ROUTE_TEMPLATES = [
|
2017-12-04 22:47:37 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.latest.{product}.{job-name}",
|
|
|
|
"index.{trust-domain}.v2.{project}.pushdate.{build_date_long}.{product}.{job-name}",
|
2019-09-05 21:11:45 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.pushdate.{build_date}.latest.{product}.{job-name}",
|
2017-12-04 22:47:37 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.pushlog-id.{pushlog_id}.{product}.{job-name}",
|
2018-03-07 22:23:00 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.revision.{branch_rev}.{product}.{job-name}",
|
2016-09-12 15:51:49 +00:00
|
|
|
]
|
|
|
|
|
2017-07-20 01:24:12 +00:00
|
|
|
# {central, inbound, autoland} write to a "trunk" index prefix. This facilitates
|
|
|
|
# walking of tasks with similar configurations.
|
|
|
|
V2_TRUNK_ROUTE_TEMPLATES = [
|
2018-03-07 22:23:00 +00:00
|
|
|
"index.{trust-domain}.v2.trunk.revision.{branch_rev}.{product}.{job-name}",
|
2017-07-20 01:24:12 +00:00
|
|
|
]
|
|
|
|
|
2017-01-09 21:23:04 +00:00
|
|
|
V2_NIGHTLY_TEMPLATES = [
|
2017-12-04 22:47:37 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.nightly.latest.{product}.{job-name}",
|
2018-03-07 22:23:00 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.nightly.{build_date}.revision.{branch_rev}.{product}.{job-name}", # noqa - too long
|
2017-12-04 22:47:37 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.nightly.{build_date}.latest.{product}.{job-name}",
|
2018-03-07 22:23:00 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.nightly.revision.{branch_rev}.{product}.{job-name}",
|
2017-01-09 21:23:04 +00:00
|
|
|
]
|
|
|
|
|
2019-03-22 03:24:44 +00:00
|
|
|
V2_SHIPPABLE_TEMPLATES = [
|
|
|
|
"index.{trust-domain}.v2.{project}.shippable.latest.{product}.{job-name}",
|
|
|
|
"index.{trust-domain}.v2.{project}.shippable.{build_date}.revision.{branch_rev}.{product}.{job-name}", # noqa - too long
|
|
|
|
"index.{trust-domain}.v2.{project}.shippable.{build_date}.latest.{product}.{job-name}",
|
|
|
|
"index.{trust-domain}.v2.{project}.shippable.revision.{branch_rev}.{product}.{job-name}",
|
|
|
|
]
|
|
|
|
|
2017-01-12 16:17:32 +00:00
|
|
|
V2_NIGHTLY_L10N_TEMPLATES = [
|
2017-12-04 22:47:37 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.nightly.latest.{product}-l10n.{job-name}.{locale}",
|
2018-03-07 22:23:00 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.nightly.{build_date}.revision.{branch_rev}.{product}-l10n.{job-name}.{locale}", # noqa - too long
|
2017-12-04 22:47:37 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.nightly.{build_date}.latest.{product}-l10n.{job-name}.{locale}", # noqa - too long
|
2018-03-07 22:23:00 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.nightly.revision.{branch_rev}.{product}-l10n.{job-name}.{locale}", # noqa - too long
|
2017-01-12 16:17:32 +00:00
|
|
|
]
|
|
|
|
|
2019-03-22 03:24:44 +00:00
|
|
|
V2_SHIPPABLE_L10N_TEMPLATES = [
|
|
|
|
"index.{trust-domain}.v2.{project}.shippable.latest.{product}-l10n.{job-name}.{locale}",
|
|
|
|
"index.{trust-domain}.v2.{project}.shippable.{build_date}.revision.{branch_rev}.{product}-l10n.{job-name}.{locale}", # noqa - too long
|
|
|
|
"index.{trust-domain}.v2.{project}.shippable.{build_date}.latest.{product}-l10n.{job-name}.{locale}", # noqa - too long
|
|
|
|
"index.{trust-domain}.v2.{project}.shippable.revision.{branch_rev}.{product}-l10n.{job-name}.{locale}", # noqa - too long
|
|
|
|
]
|
|
|
|
|
2017-01-09 21:23:04 +00:00
|
|
|
V2_L10N_TEMPLATES = [
|
2018-03-07 22:23:00 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.revision.{branch_rev}.{product}-l10n.{job-name}.{locale}",
|
2017-12-04 22:47:37 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.pushdate.{build_date_long}.{product}-l10n.{job-name}.{locale}", # noqa - too long
|
2017-12-01 16:58:29 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.pushlog-id.{pushlog_id}.{product}-l10n.{job-name}.{locale}",
|
2017-12-04 22:47:37 +00:00
|
|
|
"index.{trust-domain}.v2.{project}.latest.{product}-l10n.{job-name}.{locale}",
|
2017-01-09 21:23:04 +00:00
|
|
|
]
|
|
|
|
|
2019-04-16 09:46:07 +00:00
|
|
|
# This index is specifically for builds that include geckoview releases,
|
|
|
|
# so we can hard-code the project to "geckoview"
|
|
|
|
V2_GECKOVIEW_RELEASE = "index.{trust-domain}.v2.{project}.geckoview-version.{geckoview-version}.{product}.{job-name}" # noqa - too long
|
|
|
|
|
2017-12-08 04:19:00 +00:00
|
|
|
# the roots of the treeherder routes
|
|
|
|
TREEHERDER_ROUTE_ROOT = 'tc-treeherder'
|
2016-09-12 15:51:49 +00:00
|
|
|
|
2018-03-07 22:23:00 +00:00
|
|
|
|
|
|
|
def get_branch_rev(config):
|
2018-05-23 21:34:18 +00:00
|
|
|
return config.params['{}head_rev'.format(
|
|
|
|
config.graph_config['project-repo-param-prefix']
|
|
|
|
)]
|
|
|
|
|
|
|
|
|
|
|
|
def get_branch_repo(config):
|
|
|
|
return config.params['{}head_repository'.format(
|
|
|
|
config.graph_config['project-repo-param-prefix'],
|
|
|
|
)]
|
2018-03-07 22:23:00 +00:00
|
|
|
|
|
|
|
|
2019-03-07 04:20:36 +00:00
|
|
|
@memoize
|
|
|
|
def get_default_priority(graph_config, project):
|
|
|
|
return evaluate_keyed_by(
|
|
|
|
graph_config['task-priority'],
|
|
|
|
"Graph Config",
|
|
|
|
{'project': project}
|
|
|
|
)
|
|
|
|
|
2017-05-05 20:22:03 +00:00
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
# define a collection of payload builders, depending on the worker implementation
|
|
|
|
payload_builders = {}
|
|
|
|
|
|
|
|
|
2019-07-02 17:05:21 +00:00
|
|
|
@attr.s(frozen=True)
|
|
|
|
class PayloadBuilder(object):
|
|
|
|
schema = attr.ib(type=Schema)
|
|
|
|
builder = attr.ib()
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
def payload_builder(name, schema):
|
2019-07-12 20:44:12 +00:00
|
|
|
schema = Schema({Required('implementation'): name, Optional('os'): text_type}).extend(schema)
|
2018-10-26 19:02:36 +00:00
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
def wrap(func):
|
2019-07-02 17:05:21 +00:00
|
|
|
payload_builders[name] = PayloadBuilder(schema, func)
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
return func
|
|
|
|
return wrap
|
|
|
|
|
2017-06-15 16:10:59 +00:00
|
|
|
|
2017-01-09 21:23:04 +00:00
|
|
|
# define a collection of index builders, depending on the type implementation
|
|
|
|
index_builders = {}
|
|
|
|
|
|
|
|
|
|
|
|
def index_builder(name):
|
|
|
|
def wrap(func):
|
|
|
|
index_builders[name] = func
|
|
|
|
return func
|
|
|
|
return wrap
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2018-04-19 15:21:30 +00:00
|
|
|
UNSUPPORTED_INDEX_PRODUCT_ERROR = """\
|
2017-12-13 23:00:57 +00:00
|
|
|
The gecko-v2 product {product} is not in the list of configured products in
|
|
|
|
`taskcluster/ci/config.yml'.
|
|
|
|
"""
|
|
|
|
|
2017-12-13 23:00:14 +00:00
|
|
|
|
|
|
|
def verify_index(config, index):
|
2017-12-13 23:00:57 +00:00
|
|
|
product = index['product']
|
|
|
|
if product not in config.graph_config['index']['products']:
|
2018-04-19 15:21:30 +00:00
|
|
|
raise Exception(UNSUPPORTED_INDEX_PRODUCT_ERROR.format(product=product))
|
2017-12-13 23:00:57 +00:00
|
|
|
|
2017-11-08 04:57:35 +00:00
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('docker-worker', schema={
|
|
|
|
Required('os'): 'linux',
|
|
|
|
|
2019-02-12 21:12:27 +00:00
|
|
|
# For tasks that will run in docker-worker, this is the
|
2018-10-26 19:02:36 +00:00
|
|
|
# name of the docker image or in-tree docker image to run the task in. If
|
|
|
|
# in-tree, then a dependency will be created automatically. This is
|
|
|
|
# generally `desktop-test`, or an image that acts an awful lot like it.
|
|
|
|
Required('docker-image'): Any(
|
|
|
|
# a raw Docker image path (repo/image:tag)
|
2020-01-21 17:12:08 +00:00
|
|
|
text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
# an in-tree generated docker image (from `taskcluster/docker/<name>`)
|
2020-01-21 17:12:08 +00:00
|
|
|
{'in-tree': text_type},
|
2018-10-26 19:02:36 +00:00
|
|
|
# an indexed docker image
|
2020-01-21 17:12:08 +00:00
|
|
|
{'indexed': text_type},
|
2018-10-26 19:02:36 +00:00
|
|
|
),
|
|
|
|
|
|
|
|
# worker features that should be enabled
|
|
|
|
Required('chain-of-trust'): bool,
|
|
|
|
Required('taskcluster-proxy'): bool,
|
|
|
|
Required('allow-ptrace'): bool,
|
|
|
|
Required('loopback-video'): bool,
|
|
|
|
Required('loopback-audio'): bool,
|
|
|
|
Required('docker-in-docker'): bool, # (aka 'dind')
|
|
|
|
Required('privileged'): bool,
|
|
|
|
|
|
|
|
# Paths to Docker volumes.
|
|
|
|
#
|
|
|
|
# For in-tree Docker images, volumes can be parsed from Dockerfile.
|
|
|
|
# This only works for the Dockerfile itself: if a volume is defined in
|
|
|
|
# a base image, it will need to be declared here. Out-of-tree Docker
|
|
|
|
# images will also require explicit volume annotation.
|
|
|
|
#
|
|
|
|
# Caches are often mounted to the same path as Docker volumes. In this
|
|
|
|
# case, they take precedence over a Docker volume. But a volume still
|
|
|
|
# needs to be declared for the path.
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('volumes'): [text_type],
|
2020-03-05 23:22:13 +00:00
|
|
|
Optional(
|
|
|
|
"required-volumes",
|
|
|
|
description=(
|
|
|
|
"Paths that are required to be volumes for performance reasons. "
|
|
|
|
"For in-tree images, these paths will be checked to verify that they "
|
|
|
|
"are defined as volumes."
|
|
|
|
),
|
|
|
|
): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# caches to set up for the task
|
|
|
|
Optional('caches'): [{
|
|
|
|
# only one type is supported by any of the workers right now
|
|
|
|
'type': 'persistent',
|
|
|
|
|
|
|
|
# name of the cache, allowing re-use by subsequent tasks naming the
|
|
|
|
# same cache
|
2020-01-21 17:12:08 +00:00
|
|
|
'name': text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# location in the task image where the cache will be mounted
|
2020-01-21 17:12:08 +00:00
|
|
|
'mount-point': text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# Whether the cache is not used in untrusted environments
|
|
|
|
# (like the Try repo).
|
|
|
|
Optional('skip-untrusted'): bool,
|
|
|
|
}],
|
|
|
|
|
|
|
|
# artifacts to extract from the task image after completion
|
|
|
|
Optional('artifacts'): [{
|
|
|
|
# type of artifact -- simple file, or recursive directory
|
|
|
|
'type': Any('file', 'directory'),
|
|
|
|
|
|
|
|
# task image path from which to read artifact
|
2020-01-21 17:12:08 +00:00
|
|
|
'path': text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# name of the produced artifact (root of the names for
|
|
|
|
# type=directory)
|
2020-01-21 17:12:08 +00:00
|
|
|
'name': text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
}],
|
|
|
|
|
|
|
|
# environment variables
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('env'): {text_type: taskref_or_string},
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# the command to run; if not given, docker-worker will default to the
|
|
|
|
# command in the docker image
|
|
|
|
Optional('command'): [taskref_or_string],
|
|
|
|
|
|
|
|
# the maximum time to run, in seconds
|
|
|
|
Required('max-run-time'): int,
|
|
|
|
|
|
|
|
# the exit status code(s) that indicates the task should be retried
|
|
|
|
Optional('retry-exit-status'): [int],
|
|
|
|
|
|
|
|
# the exit status code(s) that indicates the caches used by the task
|
|
|
|
# should be purged
|
|
|
|
Optional('purge-caches-exit-status'): [int],
|
|
|
|
|
|
|
|
# Wether any artifacts are assigned to this worker
|
|
|
|
Optional('skip-artifacts'): bool,
|
|
|
|
})
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
def build_docker_worker_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
2017-08-22 22:06:40 +00:00
|
|
|
level = int(config.params['level'])
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
image = worker['docker-image']
|
|
|
|
if isinstance(image, dict):
|
2017-07-27 20:39:51 +00:00
|
|
|
if 'in-tree' in image:
|
2017-08-23 15:53:56 +00:00
|
|
|
name = image['in-tree']
|
2017-07-27 20:39:51 +00:00
|
|
|
docker_image_task = 'build-docker-image-' + image['in-tree']
|
|
|
|
task.setdefault('dependencies', {})['docker-image'] = docker_image_task
|
|
|
|
|
|
|
|
image = {
|
|
|
|
"path": "public/image.tar.zst",
|
|
|
|
"taskId": {"task-reference": "<docker-image>"},
|
|
|
|
"type": "task-image",
|
|
|
|
}
|
2017-08-23 15:53:56 +00:00
|
|
|
|
|
|
|
# Find VOLUME in Dockerfile.
|
|
|
|
volumes = dockerutil.parse_volumes(name)
|
|
|
|
for v in sorted(volumes):
|
|
|
|
if v in worker['volumes']:
|
|
|
|
raise Exception('volume %s already defined; '
|
|
|
|
'if it is defined in a Dockerfile, '
|
|
|
|
'it does not need to be specified in the '
|
|
|
|
'worker definition' % v)
|
|
|
|
|
|
|
|
worker['volumes'].append(v)
|
|
|
|
|
2017-07-27 20:39:51 +00:00
|
|
|
elif 'indexed' in image:
|
|
|
|
image = {
|
|
|
|
"path": "public/image.tar.zst",
|
|
|
|
"namespace": image['indexed'],
|
|
|
|
"type": "indexed-image",
|
|
|
|
}
|
|
|
|
else:
|
|
|
|
raise Exception("unknown docker image type")
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
|
|
|
features = {}
|
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
if worker.get('taskcluster-proxy'):
|
|
|
|
features['taskclusterProxy'] = True
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
if worker.get('allow-ptrace'):
|
|
|
|
features['allowPtrace'] = True
|
|
|
|
task_def['scopes'].append('docker-worker:feature:allowPtrace')
|
|
|
|
|
2016-10-28 14:36:09 +00:00
|
|
|
if worker.get('chain-of-trust'):
|
2016-10-04 21:04:21 +00:00
|
|
|
features['chainOfTrust'] = True
|
|
|
|
|
2017-03-08 20:52:13 +00:00
|
|
|
if worker.get('docker-in-docker'):
|
|
|
|
features['dind'] = True
|
|
|
|
|
2016-11-10 18:02:54 +00:00
|
|
|
if task.get('needs-sccache'):
|
|
|
|
features['taskclusterProxy'] = True
|
|
|
|
task_def['scopes'].append(
|
2017-12-21 21:43:38 +00:00
|
|
|
'assume:project:taskcluster:{trust_domain}:level-{level}-sccache-buckets'.format(
|
|
|
|
trust_domain=config.graph_config['trust-domain'],
|
2016-11-10 18:02:54 +00:00
|
|
|
level=config.params['level'])
|
|
|
|
)
|
|
|
|
worker['env']['USE_SCCACHE'] = '1'
|
2019-09-09 16:51:42 +00:00
|
|
|
worker['env']['SCCACHE_GCS_PROJECT'] = SCCACHE_GCS_PROJECT
|
2018-07-02 23:17:41 +00:00
|
|
|
# Disable sccache idle shutdown.
|
|
|
|
worker['env']['SCCACHE_IDLE_TIMEOUT'] = '0'
|
2017-01-27 08:21:32 +00:00
|
|
|
else:
|
|
|
|
worker['env']['SCCACHE_DISABLE'] = '1'
|
2016-11-10 18:02:54 +00:00
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
capabilities = {}
|
|
|
|
|
|
|
|
for lo in 'audio', 'video':
|
|
|
|
if worker.get('loopback-' + lo):
|
|
|
|
capitalized = 'loopback' + lo.capitalize()
|
|
|
|
devices = capabilities.setdefault('devices', {})
|
|
|
|
devices[capitalized] = True
|
|
|
|
task_def['scopes'].append('docker-worker:capability:device:' + capitalized)
|
|
|
|
|
2018-08-07 16:05:38 +00:00
|
|
|
if worker.get('privileged'):
|
|
|
|
capabilities['privileged'] = True
|
|
|
|
task_def['scopes'].append('docker-worker:capability:privileged')
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
task_def['payload'] = payload = {
|
|
|
|
'image': image,
|
|
|
|
'env': worker['env'],
|
|
|
|
}
|
2017-03-08 20:52:13 +00:00
|
|
|
if 'command' in worker:
|
|
|
|
payload['command'] = worker['command']
|
2016-09-12 15:51:49 +00:00
|
|
|
|
|
|
|
if 'max-run-time' in worker:
|
|
|
|
payload['maxRunTime'] = worker['max-run-time']
|
|
|
|
|
2018-06-19 20:58:29 +00:00
|
|
|
run_task = payload.get('command', [''])[0].endswith('run-task')
|
|
|
|
|
|
|
|
# run-task exits EXIT_PURGE_CACHES if there is a problem with caches.
|
|
|
|
# Automatically retry the tasks and purge caches if we see this exit
|
|
|
|
# code.
|
|
|
|
# TODO move this closer to code adding run-task once bug 1469697 is
|
|
|
|
# addressed.
|
|
|
|
if run_task:
|
|
|
|
worker.setdefault('retry-exit-status', []).append(72)
|
|
|
|
worker.setdefault('purge-caches-exit-status', []).append(72)
|
|
|
|
|
2018-01-16 05:00:50 +00:00
|
|
|
payload['onExitStatus'] = {}
|
2016-10-07 16:20:15 +00:00
|
|
|
if 'retry-exit-status' in worker:
|
2018-01-16 05:00:50 +00:00
|
|
|
payload['onExitStatus']['retry'] = worker['retry-exit-status']
|
|
|
|
if 'purge-caches-exit-status' in worker:
|
|
|
|
payload['onExitStatus']['purgeCaches'] = worker['purge-caches-exit-status']
|
2016-10-07 16:20:15 +00:00
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
if 'artifacts' in worker:
|
|
|
|
artifacts = {}
|
|
|
|
for artifact in worker['artifacts']:
|
|
|
|
artifacts[artifact['name']] = {
|
|
|
|
'path': artifact['path'],
|
|
|
|
'type': artifact['type'],
|
|
|
|
'expires': task_def['expires'], # always expire with the task
|
|
|
|
}
|
|
|
|
payload['artifacts'] = artifacts
|
|
|
|
|
2020-01-21 17:12:08 +00:00
|
|
|
if isinstance(worker.get('docker-image'), text_type):
|
2017-09-06 22:35:13 +00:00
|
|
|
out_of_tree_image = worker['docker-image']
|
Bug 1430037 - Use the in-tree image builder image to build docker images. r=dustin
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 978cf033732cbbbb277d206dec69660175b82afa
2018-01-11 08:50:00 +00:00
|
|
|
run_task = run_task or out_of_tree_image.startswith(
|
|
|
|
'taskcluster/image_builder')
|
2017-09-06 22:35:13 +00:00
|
|
|
else:
|
|
|
|
out_of_tree_image = None
|
Bug 1430037 - Use the in-tree image builder image to build docker images. r=dustin
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 978cf033732cbbbb277d206dec69660175b82afa
2018-01-11 08:50:00 +00:00
|
|
|
image = worker.get('docker-image', {}).get('in-tree')
|
|
|
|
run_task = run_task or image == 'image_builder'
|
2017-08-23 17:47:37 +00:00
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
if 'caches' in worker:
|
|
|
|
caches = {}
|
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-18 21:07:03 +00:00
|
|
|
|
|
|
|
# run-task knows how to validate caches.
|
|
|
|
#
|
|
|
|
# To help ensure new run-task features and bug fixes don't interfere
|
|
|
|
# with existing caches, we seed the hash of run-task into cache names.
|
|
|
|
# So, any time run-task changes, we should get a fresh set of caches.
|
|
|
|
# This means run-task can make changes to cache interaction at any time
|
|
|
|
# without regards for backwards or future compatibility.
|
2017-09-06 23:09:15 +00:00
|
|
|
#
|
|
|
|
# But this mechanism only works for in-tree Docker images that are built
|
|
|
|
# with the current run-task! For out-of-tree Docker images, we have no
|
|
|
|
# way of knowing their content of run-task. So, in addition to varying
|
|
|
|
# cache names by the contents of run-task, we also take the Docker image
|
|
|
|
# name into consideration. This means that different Docker images will
|
|
|
|
# never share the same cache. This is a bit unfortunate. But it is the
|
|
|
|
# safest thing to do. Fortunately, most images are defined in-tree.
|
|
|
|
#
|
|
|
|
# For out-of-tree Docker images, we don't strictly need to incorporate
|
|
|
|
# the run-task content into the cache name. However, doing so preserves
|
|
|
|
# the mechanism whereby changing run-task results in new caches
|
|
|
|
# everywhere.
|
2018-01-23 06:08:33 +00:00
|
|
|
|
|
|
|
# As an additional mechanism to force the use of different caches, the
|
|
|
|
# string literal in the variable below can be changed. This is
|
|
|
|
# preferred to changing run-task because it doesn't require images
|
|
|
|
# to be rebuilt.
|
2018-01-02 21:22:36 +00:00
|
|
|
cache_version = 'v3'
|
2018-01-23 06:08:33 +00:00
|
|
|
|
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-18 21:07:03 +00:00
|
|
|
if run_task:
|
2019-02-21 07:32:36 +00:00
|
|
|
suffix = '{}-{}'.format(cache_version, _run_task_suffix())
|
2017-09-06 23:09:15 +00:00
|
|
|
|
|
|
|
if out_of_tree_image:
|
2020-02-07 18:12:23 +00:00
|
|
|
name_hash = hashlib.sha256(
|
|
|
|
six.ensure_binary(out_of_tree_image)).hexdigest()
|
2017-09-06 23:09:15 +00:00
|
|
|
suffix += name_hash[0:12]
|
|
|
|
|
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-18 21:07:03 +00:00
|
|
|
else:
|
2019-02-21 07:32:36 +00:00
|
|
|
suffix = cache_version
|
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-18 21:07:03 +00:00
|
|
|
|
2018-01-16 07:15:59 +00:00
|
|
|
skip_untrusted = config.params.is_try() or level == 1
|
2017-08-22 22:06:40 +00:00
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
for cache in worker['caches']:
|
2017-08-22 22:06:40 +00:00
|
|
|
# Some caches aren't enabled in environments where we can't
|
|
|
|
# guarantee certain behavior. Filter those out.
|
|
|
|
if cache.get('skip-untrusted') and skip_untrusted:
|
|
|
|
continue
|
|
|
|
|
2019-02-21 07:32:36 +00:00
|
|
|
name = '{trust_domain}-level-{level}-{name}-{suffix}'.format(
|
|
|
|
trust_domain=config.graph_config['trust-domain'],
|
|
|
|
level=config.params['level'],
|
|
|
|
name=cache['name'],
|
|
|
|
suffix=suffix,
|
|
|
|
)
|
|
|
|
|
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-18 21:07:03 +00:00
|
|
|
caches[name] = cache['mount-point']
|
|
|
|
task_def['scopes'].append('docker-worker:cache:%s' % name)
|
|
|
|
|
|
|
|
# Assertion: only run-task is interested in this.
|
|
|
|
if run_task:
|
|
|
|
payload['env']['TASKCLUSTER_CACHES'] = ';'.join(sorted(
|
|
|
|
caches.values()))
|
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
payload['cache'] = caches
|
|
|
|
|
2017-08-23 17:47:37 +00:00
|
|
|
# And send down volumes information to run-task as well.
|
|
|
|
if run_task and worker.get('volumes'):
|
|
|
|
payload['env']['TASKCLUSTER_VOLUMES'] = ';'.join(
|
2020-02-07 18:12:23 +00:00
|
|
|
[six.ensure_text(s) for s in sorted(worker['volumes'])])
|
2017-08-23 17:47:37 +00:00
|
|
|
|
Bug 1391476 - Add UID and GID to cache parameters; r=dustin
The UID and GID that a task executes under is dynamic. As a result,
caches need to be aware of the UID and GID that owns files otherwise
subsequent tasks could run into permission denied errors. This is
why `run-task --chown-recursive` exists. By recursively changing
ownership of persisted files, we ensure the current task is able
to read and write all existing files.
When you take a step back, you realize that chowning of cached
files is an expensive workaround. Yes, this results in cache hits.
But the cost is you potentially have to perform hundreds of thousands
of I/O system calls to mass chown. The ideal situation is that
UID/GID is consistent across tasks on any given cache and
potentially expensive permissions setting can be avoided. So, that's
what this commit does.
We add the task's UID and GID to run-task's requirements. When we
first see a cache, we record a UID and GID with it and chown the
empty cache directory to that UID and GID. Subsequent tasks using
this cache *must* use the same UID and GID or else run-task will
fail.
Since run-task now guarantees that all cache consumers use the same
UID and GID, we can avoid a potentially expensive recursive chown.
But there is an exception. In untrusted environments (namely Try),
we recursively chown existing caches if there is a uid/gid mismatch.
We do this because Try is a sandbox and any random task could
experiment with a non-standard uid/gid. That populated cache would
"poison" the cache for the next caller. Or vice-versa. It would be
annoying if caches were randomly poisoned due to Try pushes that
didn't realize there was a UID/GID mismatch. We could outlaw "bad"
UID and GIDs. But that makes the barrier to testing things on Try
harder. So, we go with the flow and recursively chown caches in
this scenario.
This change will shine light on all tasks using inconsistent UID
and GID values on the same cache. Bustage is anticipated.
Unfortunately, we can't easily know what will break. So it will be
one of those things where we will have to fix problems as they arise.
Fortunately, because caches are now tied to the content of run-task,
we only need to back out this change and tasks should revert to caches
without UID and GID pinning requirements and everything will work
again.
MozReview-Commit-ID: 2ka4rOnnXIp
--HG--
extra : rebase_source : ccb2b0a9230694f989775b26d5276fd3ac928af3
extra : source : 083d2e1cc8fe44b04e44f74bda3dd8bc75ba826c
2017-08-22 23:49:26 +00:00
|
|
|
if payload.get('cache') and skip_untrusted:
|
|
|
|
payload['env']['TASKCLUSTER_UNTRUSTED_CACHES'] = '1'
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
if features:
|
|
|
|
payload['features'] = features
|
|
|
|
if capabilities:
|
|
|
|
payload['capabilities'] = capabilities
|
|
|
|
|
2017-08-23 15:57:59 +00:00
|
|
|
check_caches_are_volumes(task)
|
2020-03-05 23:22:13 +00:00
|
|
|
check_required_volumes(task)
|
2017-08-23 15:57:59 +00:00
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('generic-worker', schema={
|
2019-04-15 17:30:59 +00:00
|
|
|
Required('os'): Any('windows', 'macosx', 'linux', 'linux-bitbar'),
|
2018-10-26 19:02:36 +00:00
|
|
|
# see http://schemas.taskcluster.net/generic-worker/v1/payload.json
|
|
|
|
# and https://docs.taskcluster.net/reference/workers/generic-worker/payload
|
|
|
|
|
|
|
|
# command is a list of commands to run, sequentially
|
|
|
|
# on Windows, each command is a string, on OS X and Linux, each command is
|
|
|
|
# a string array
|
|
|
|
Required('command'): Any(
|
|
|
|
[taskref_or_string], # Windows
|
|
|
|
[[taskref_or_string]] # Linux / OS X
|
|
|
|
),
|
|
|
|
|
|
|
|
# artifacts to extract from the task image after completion; note that artifacts
|
|
|
|
# for the generic worker cannot have names
|
|
|
|
Optional('artifacts'): [{
|
|
|
|
# type of artifact -- simple file, or recursive directory
|
|
|
|
'type': Any('file', 'directory'),
|
|
|
|
|
|
|
|
# filesystem path from which to read artifact
|
2020-01-21 17:12:08 +00:00
|
|
|
'path': text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# if not specified, path is used for artifact name
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('name'): text_type
|
2018-10-26 19:02:36 +00:00
|
|
|
}],
|
|
|
|
|
|
|
|
# Directories and/or files to be mounted.
|
|
|
|
# The actual allowed combinations are stricter than the model below,
|
|
|
|
# but this provides a simple starting point.
|
|
|
|
# See https://docs.taskcluster.net/reference/workers/generic-worker/payload
|
|
|
|
Optional('mounts'): [{
|
|
|
|
# A unique name for the cache volume, implies writable cache directory
|
|
|
|
# (otherwise mount is a read-only file or directory).
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('cache-name'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
# Optional content for pre-loading cache, or mandatory content for
|
|
|
|
# read-only file or directory. Pre-loaded content can come from either
|
|
|
|
# a task artifact or from a URL.
|
|
|
|
Optional('content'): {
|
|
|
|
|
|
|
|
# *** Either (artifact and task-id) or url must be specified. ***
|
|
|
|
|
|
|
|
# Artifact name that contains the content.
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('artifact'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
# Task ID that has the artifact that contains the content.
|
|
|
|
Optional('task-id'): taskref_or_string,
|
|
|
|
# URL that supplies the content in response to an unauthenticated
|
|
|
|
# GET request.
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('url'): text_type
|
2018-10-26 19:02:36 +00:00
|
|
|
},
|
|
|
|
|
|
|
|
# *** Either file or directory must be specified. ***
|
|
|
|
|
|
|
|
# If mounting a cache or read-only directory, the filesystem location of
|
|
|
|
# the directory should be specified as a relative path to the task
|
|
|
|
# directory here.
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('directory'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
# If mounting a file, specify the relative path within the task
|
|
|
|
# directory to mount the file (the file will be read only).
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('file'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
# Required if and only if `content` is specified and mounting a
|
|
|
|
# directory (not a file). This should be the archive format of the
|
|
|
|
# content (either pre-loaded cache or read-only directory).
|
|
|
|
Optional('format'): Any('rar', 'tar.bz2', 'tar.gz', 'zip')
|
|
|
|
}],
|
|
|
|
|
|
|
|
# environment variables
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('env'): {text_type: taskref_or_string},
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# the maximum time to run, in seconds
|
|
|
|
Required('max-run-time'): int,
|
|
|
|
|
|
|
|
# os user groups for test task workers
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('os-groups'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# feature for test task to run as administarotr
|
|
|
|
Optional('run-as-administrator'): bool,
|
|
|
|
|
|
|
|
# optional features
|
|
|
|
Required('chain-of-trust'): bool,
|
|
|
|
Optional('taskcluster-proxy'): bool,
|
|
|
|
|
2019-10-17 01:30:17 +00:00
|
|
|
# the exit status code(s) that indicates the task should be retried
|
|
|
|
Optional('retry-exit-status'): [int],
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
# Wether any artifacts are assigned to this worker
|
|
|
|
Optional('skip-artifacts'): bool,
|
|
|
|
})
|
2016-07-29 17:50:09 +00:00
|
|
|
def build_generic_worker_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
2019-08-23 12:39:25 +00:00
|
|
|
features = {}
|
2016-07-29 17:50:09 +00:00
|
|
|
|
2018-08-29 17:13:40 +00:00
|
|
|
task_def['payload'] = {
|
|
|
|
'command': worker['command'],
|
|
|
|
'maxRunTime': worker['max-run-time'],
|
|
|
|
}
|
|
|
|
|
2019-05-08 06:00:00 +00:00
|
|
|
if worker['os'] == 'windows':
|
|
|
|
task_def['payload']['onExitStatus'] = {
|
|
|
|
'retry': [
|
|
|
|
# These codes (on windows) indicate a process interruption,
|
|
|
|
# rather than a task run failure. See bug 1544403.
|
|
|
|
1073807364, # process force-killed due to system shutdown
|
|
|
|
3221225786, # sigint (any interrupt)
|
|
|
|
]
|
|
|
|
}
|
2019-10-17 01:30:17 +00:00
|
|
|
if 'retry-exit-status' in worker:
|
|
|
|
task_def['payload'].setdefault(
|
|
|
|
'onExitStatus', {}).setdefault('retry', []).extend(worker['retry-exit-status'])
|
2020-01-23 20:21:27 +00:00
|
|
|
if worker['os'] == 'linux-bitbar':
|
|
|
|
task_def['payload'].setdefault('onExitStatus', {}).setdefault('retry', [])
|
|
|
|
# exit code 4 is used to indicate an intermittent android device error
|
|
|
|
if 4 not in task_def['payload']['onExitStatus']['retry']:
|
|
|
|
task_def['payload']['onExitStatus']['retry'].extend([4])
|
2019-05-08 06:00:00 +00:00
|
|
|
|
2018-08-29 17:13:40 +00:00
|
|
|
env = worker.get('env', {})
|
|
|
|
|
|
|
|
if task.get('needs-sccache'):
|
2019-08-23 12:39:25 +00:00
|
|
|
features['taskclusterProxy'] = True
|
|
|
|
task_def['scopes'].append(
|
|
|
|
'assume:project:taskcluster:{trust_domain}:level-{level}-sccache-buckets'.format(
|
|
|
|
trust_domain=config.graph_config['trust-domain'],
|
|
|
|
level=config.params['level'])
|
|
|
|
)
|
2018-08-29 17:13:40 +00:00
|
|
|
env['USE_SCCACHE'] = '1'
|
2019-09-09 16:51:42 +00:00
|
|
|
worker['env']['SCCACHE_GCS_PROJECT'] = SCCACHE_GCS_PROJECT
|
2018-08-29 17:13:40 +00:00
|
|
|
# Disable sccache idle shutdown.
|
|
|
|
env['SCCACHE_IDLE_TIMEOUT'] = '0'
|
|
|
|
else:
|
|
|
|
env['SCCACHE_DISABLE'] = '1'
|
|
|
|
|
|
|
|
if env:
|
|
|
|
task_def['payload']['env'] = env
|
|
|
|
|
2016-07-29 17:50:09 +00:00
|
|
|
artifacts = []
|
|
|
|
|
2018-06-26 15:06:13 +00:00
|
|
|
for artifact in worker.get('artifacts', []):
|
2017-04-06 16:50:41 +00:00
|
|
|
a = {
|
2016-07-29 17:50:09 +00:00
|
|
|
'path': artifact['path'],
|
|
|
|
'type': artifact['type'],
|
2017-04-06 16:50:41 +00:00
|
|
|
}
|
|
|
|
if 'name' in artifact:
|
|
|
|
a['name'] = artifact['name']
|
|
|
|
artifacts.append(a)
|
|
|
|
|
2018-08-29 17:13:40 +00:00
|
|
|
if artifacts:
|
|
|
|
task_def['payload']['artifacts'] = artifacts
|
|
|
|
|
2017-04-06 16:50:41 +00:00
|
|
|
# Need to copy over mounts, but rename keys to respect naming convention
|
|
|
|
# * 'cache-name' -> 'cacheName'
|
|
|
|
# * 'task-id' -> 'taskId'
|
|
|
|
# All other key names are already suitable, and don't need renaming.
|
|
|
|
mounts = deepcopy(worker.get('mounts', []))
|
|
|
|
for mount in mounts:
|
|
|
|
if 'cache-name' in mount:
|
2019-02-21 07:32:36 +00:00
|
|
|
mount['cacheName'] = '{trust_domain}-level-{level}-{name}'.format(
|
|
|
|
trust_domain=config.graph_config['trust-domain'],
|
|
|
|
level=config.params['level'],
|
|
|
|
name=mount.pop('cache-name'),
|
|
|
|
)
|
2018-10-23 07:53:34 +00:00
|
|
|
task_def['scopes'].append('generic-worker:cache:{}'.format(mount['cacheName']))
|
2017-04-06 16:50:41 +00:00
|
|
|
if 'content' in mount:
|
|
|
|
if 'task-id' in mount['content']:
|
|
|
|
mount['content']['taskId'] = mount['content'].pop('task-id')
|
2018-10-23 07:53:34 +00:00
|
|
|
if 'artifact' in mount['content']:
|
|
|
|
if not mount['content']['artifact'].startswith('public/'):
|
|
|
|
task_def['scopes'].append(
|
|
|
|
'queue:get-artifact:{}'.format(mount['content']['artifact']))
|
2016-10-26 20:18:24 +00:00
|
|
|
|
2018-08-29 17:13:40 +00:00
|
|
|
if mounts:
|
|
|
|
task_def['payload']['mounts'] = mounts
|
2016-07-29 17:50:09 +00:00
|
|
|
|
2019-03-08 07:00:02 +00:00
|
|
|
if worker.get('os-groups'):
|
2018-08-29 17:13:40 +00:00
|
|
|
task_def['payload']['osGroups'] = worker['os-groups']
|
2019-03-08 07:00:02 +00:00
|
|
|
task_def['scopes'].extend(
|
|
|
|
['generic-worker:os-group:{}/{}'.format(
|
|
|
|
task['worker-type'],
|
|
|
|
group
|
|
|
|
) for group in worker['os-groups']])
|
2016-11-10 18:02:54 +00:00
|
|
|
|
2017-04-06 16:50:41 +00:00
|
|
|
if worker.get('chain-of-trust'):
|
|
|
|
features['chainOfTrust'] = True
|
|
|
|
|
2018-04-18 17:02:02 +00:00
|
|
|
if worker.get('taskcluster-proxy'):
|
|
|
|
features['taskclusterProxy'] = True
|
|
|
|
|
2018-09-11 19:44:08 +00:00
|
|
|
if worker.get('run-as-administrator', False):
|
|
|
|
features['runAsAdministrator'] = True
|
2019-03-08 07:00:02 +00:00
|
|
|
task_def['scopes'].append(
|
|
|
|
'generic-worker:run-as-administrator:{}'.format(task['worker-type']),
|
|
|
|
)
|
2018-09-11 19:44:08 +00:00
|
|
|
|
2017-04-06 16:50:41 +00:00
|
|
|
if features:
|
|
|
|
task_def['payload']['features'] = features
|
|
|
|
|
2016-07-29 17:50:09 +00:00
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('scriptworker-signing', schema={
|
|
|
|
# the maximum time to run, in seconds
|
|
|
|
Required('max-run-time'): int,
|
|
|
|
|
|
|
|
# list of artifact URLs for the artifacts that should be signed
|
|
|
|
Required('upstream-artifacts'): [{
|
|
|
|
# taskId of the task with the artifact
|
|
|
|
Required('taskId'): taskref_or_string,
|
|
|
|
|
|
|
|
# type of signing task (for CoT)
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('taskType'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# Paths to the artifacts to sign
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('paths'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# Signing formats to use on each of the paths
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('formats'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
}],
|
2019-06-05 22:02:25 +00:00
|
|
|
|
|
|
|
# behavior for mac iscript
|
|
|
|
Optional('mac-behavior'): Any(
|
2020-02-25 15:38:27 +00:00
|
|
|
"mac_notarize_part_1", "mac_notarize_part_3", "mac_sign_and_pkg",
|
|
|
|
"mac_geckodriver",
|
2019-06-05 22:02:25 +00:00
|
|
|
),
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('entitlements-url'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
})
|
Bug 1322041 - Add signing support for all nightlies. r=dustin
Involved work which landed on the date project branch with:
* Bug 1277579, by kmoir@mozilla.com (https://hg.mozilla.org/projects/date/rev/afd3823c852b,
https://hg.mozilla.org/projects/date/rev/345f83708453,
https://hg.mozilla.org/projects/date/rev/8a3ed233af86,
https://hg.mozilla.org/projects/date/rev/0d737cf8e743,
https://hg.mozilla.org/projects/date/rev/e42b3d606002,
https://hg.mozilla.org/projects/date/rev/c3a160ac642b)
* Bug 1305096, by kmoir@mozilla.com (https://hg.mozilla.org/projects/date/rev/2287a2568038)
* Bug 1277579, by dustin@mozilla.com (https://hg.mozilla.org/projects/date/rev/79ceb7fa0589)
* Bug 1306166, by kmoir@mozilla.com (https://hg.mozilla.org/projects/date/rev/d3fd1966095d,
https://hg.mozilla.org/projects/date/rev/2f52061c02e6,
https://hg.mozilla.org/projects/date/rev/a9f52549a3c3,
https://hg.mozilla.org/projects/date/rev/f69efa90945a)
* Bug 1314847, by Callek@gmail.com (https://hg.mozilla.org/projects/date/rev/4a1231655fbb,
https://hg.mozilla.org/projects/date/rev/3cd3a0d32f43)
* Bug 1312000, by Callek@gmail.com (https://hg.mozilla.org/projects/date/rev/79a2f66ff5c0)
* Bug 1312500, by Callek@gmail.com (https://hg.mozilla.org/projects/date/rev/6ae07fa4b011)
* Bug 1316214, by Callek@gmail.com (https://hg.mozilla.org/projects/date/rev/edae37481cab,
https://hg.mozilla.org/projects/date/rev/62bd3371e954)
* Bug 1319189, by Callek@gmail.com (https://hg.mozilla.org/projects/date/rev/4c33f8ccecf5)
* Bug 1319546, by kmoir@mozilla.com (https://hg.mozilla.org/projects/date/rev/70a23d243d2c)
* No Bug, by asasaki@mozilla.com (https://hg.mozilla.org/projects/date/rev/5d8ba3560ae9)
* No Bug, by kmoir@mozilla.com (https://hg.mozilla.org/projects/date/rev/37d9733a7174)
MozReview-Commit-ID: K1uOY4HOWPX
--HG--
extra : rebase_source : 567392d5d5ddb5ee638c53221a6e545e7b5f1805
2016-12-21 16:21:06 +00:00
|
|
|
def build_scriptworker_signing_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'maxRunTime': worker['max-run-time'],
|
|
|
|
'upstreamArtifacts': worker['upstream-artifacts']
|
|
|
|
}
|
2019-06-05 22:02:25 +00:00
|
|
|
if worker.get('mac-behavior'):
|
|
|
|
task_def['payload']['behavior'] = worker['mac-behavior']
|
2019-06-05 22:03:36 +00:00
|
|
|
if worker.get('entitlements-url'):
|
|
|
|
task_def['payload']['entitlements-url'] = worker['entitlements-url']
|
2018-08-20 18:29:58 +00:00
|
|
|
artifacts = set(task.get('release-artifacts', []))
|
|
|
|
for upstream_artifact in worker['upstream-artifacts']:
|
|
|
|
for path in upstream_artifact['paths']:
|
|
|
|
artifacts.update(get_signed_artifacts(
|
|
|
|
input=path,
|
|
|
|
formats=upstream_artifact['formats'],
|
2019-06-05 22:02:25 +00:00
|
|
|
behavior=worker.get('mac-behavior'),
|
2018-08-20 18:29:58 +00:00
|
|
|
))
|
|
|
|
task['release-artifacts'] = list(artifacts)
|
|
|
|
|
Bug 1322041 - Add signing support for all nightlies. r=dustin
Involved work which landed on the date project branch with:
* Bug 1277579, by kmoir@mozilla.com (https://hg.mozilla.org/projects/date/rev/afd3823c852b,
https://hg.mozilla.org/projects/date/rev/345f83708453,
https://hg.mozilla.org/projects/date/rev/8a3ed233af86,
https://hg.mozilla.org/projects/date/rev/0d737cf8e743,
https://hg.mozilla.org/projects/date/rev/e42b3d606002,
https://hg.mozilla.org/projects/date/rev/c3a160ac642b)
* Bug 1305096, by kmoir@mozilla.com (https://hg.mozilla.org/projects/date/rev/2287a2568038)
* Bug 1277579, by dustin@mozilla.com (https://hg.mozilla.org/projects/date/rev/79ceb7fa0589)
* Bug 1306166, by kmoir@mozilla.com (https://hg.mozilla.org/projects/date/rev/d3fd1966095d,
https://hg.mozilla.org/projects/date/rev/2f52061c02e6,
https://hg.mozilla.org/projects/date/rev/a9f52549a3c3,
https://hg.mozilla.org/projects/date/rev/f69efa90945a)
* Bug 1314847, by Callek@gmail.com (https://hg.mozilla.org/projects/date/rev/4a1231655fbb,
https://hg.mozilla.org/projects/date/rev/3cd3a0d32f43)
* Bug 1312000, by Callek@gmail.com (https://hg.mozilla.org/projects/date/rev/79a2f66ff5c0)
* Bug 1312500, by Callek@gmail.com (https://hg.mozilla.org/projects/date/rev/6ae07fa4b011)
* Bug 1316214, by Callek@gmail.com (https://hg.mozilla.org/projects/date/rev/edae37481cab,
https://hg.mozilla.org/projects/date/rev/62bd3371e954)
* Bug 1319189, by Callek@gmail.com (https://hg.mozilla.org/projects/date/rev/4c33f8ccecf5)
* Bug 1319546, by kmoir@mozilla.com (https://hg.mozilla.org/projects/date/rev/70a23d243d2c)
* No Bug, by asasaki@mozilla.com (https://hg.mozilla.org/projects/date/rev/5d8ba3560ae9)
* No Bug, by kmoir@mozilla.com (https://hg.mozilla.org/projects/date/rev/37d9733a7174)
MozReview-Commit-ID: K1uOY4HOWPX
--HG--
extra : rebase_source : 567392d5d5ddb5ee638c53221a6e545e7b5f1805
2016-12-21 16:21:06 +00:00
|
|
|
|
2020-02-25 15:40:55 +00:00
|
|
|
@payload_builder('notarization-poller', schema={
|
|
|
|
Required('uuid-manifest'): taskref_or_string,
|
|
|
|
})
|
|
|
|
def notarization_poller_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
|
|
|
task_def['payload'] = {
|
|
|
|
'uuid_manifest': worker['uuid-manifest']
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('beetmover', schema={
|
|
|
|
# the maximum time to run, in seconds
|
|
|
|
Required('max-run-time', default=600): int,
|
|
|
|
|
|
|
|
# locale key, if this is a locale beetmover job
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('locale'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
Optional('partner-public'): bool,
|
|
|
|
|
|
|
|
Required('release-properties'): {
|
2020-01-21 17:12:08 +00:00
|
|
|
'app-name': text_type,
|
|
|
|
'app-version': text_type,
|
|
|
|
'branch': text_type,
|
|
|
|
'build-id': text_type,
|
|
|
|
'hash-type': text_type,
|
|
|
|
'platform': text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
},
|
|
|
|
|
|
|
|
# list of artifact URLs for the artifacts that should be beetmoved
|
|
|
|
Required('upstream-artifacts'): [{
|
|
|
|
# taskId of the task with the artifact
|
|
|
|
Required('taskId'): taskref_or_string,
|
|
|
|
|
|
|
|
# type of signing task (for CoT)
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('taskType'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# Paths to the artifacts to sign
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('paths'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# locale is used to map upload path and allow for duplicate simple names
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('locale'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
}],
|
2018-11-28 11:13:52 +00:00
|
|
|
Optional('artifact-map'): object,
|
2018-10-26 19:02:36 +00:00
|
|
|
})
|
2017-01-12 22:45:36 +00:00
|
|
|
def build_beetmover_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
2017-03-02 01:49:57 +00:00
|
|
|
release_config = get_release_config(config)
|
2018-02-15 13:46:19 +00:00
|
|
|
release_properties = worker['release-properties']
|
2017-01-12 22:45:36 +00:00
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'maxRunTime': worker['max-run-time'],
|
2018-02-15 13:46:19 +00:00
|
|
|
'releaseProperties': {
|
|
|
|
'appName': release_properties['app-name'],
|
|
|
|
'appVersion': release_properties['app-version'],
|
|
|
|
'branch': release_properties['branch'],
|
|
|
|
'buildid': release_properties['build-id'],
|
|
|
|
'hashType': release_properties['hash-type'],
|
|
|
|
'platform': release_properties['platform'],
|
|
|
|
},
|
2017-01-12 22:45:36 +00:00
|
|
|
'upload_date': config.params['build_date'],
|
2018-02-15 13:46:19 +00:00
|
|
|
'upstreamArtifacts': worker['upstream-artifacts'],
|
2017-01-12 22:45:36 +00:00
|
|
|
}
|
|
|
|
if worker.get('locale'):
|
|
|
|
task_def['payload']['locale'] = worker['locale']
|
2018-11-28 11:13:52 +00:00
|
|
|
if worker.get('artifact-map'):
|
|
|
|
task_def['payload']['artifactMap'] = worker['artifact-map']
|
2018-04-18 17:02:02 +00:00
|
|
|
if worker.get('partner-public'):
|
|
|
|
task_def['payload']['is_partner_repack_public'] = worker['partner-public']
|
2017-03-02 01:49:57 +00:00
|
|
|
if release_config:
|
|
|
|
task_def['payload'].update(release_config)
|
2017-01-12 22:45:36 +00:00
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('beetmover-push-to-release', schema={
|
|
|
|
# the maximum time to run, in seconds
|
|
|
|
Required('max-run-time'): int,
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('product'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
})
|
2018-05-04 15:37:46 +00:00
|
|
|
def build_beetmover_push_to_release_payload(config, task, task_def):
|
2017-10-30 00:32:22 +00:00
|
|
|
worker = task['worker']
|
2017-12-06 03:35:45 +00:00
|
|
|
release_config = get_release_config(config)
|
2019-06-21 22:12:59 +00:00
|
|
|
partners = ['{}/{}'.format(p, s) for p, s, _ in get_partners_to_be_published(config)]
|
2017-10-30 00:32:22 +00:00
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'maxRunTime': worker['max-run-time'],
|
|
|
|
'product': worker['product'],
|
|
|
|
'version': release_config['version'],
|
|
|
|
'build_number': release_config['build_number'],
|
2019-06-21 22:12:59 +00:00
|
|
|
'partners': partners,
|
2017-10-30 00:32:22 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('beetmover-maven', schema={
|
|
|
|
Required('max-run-time', default=600): int,
|
|
|
|
Required('release-properties'): {
|
2020-01-21 17:12:08 +00:00
|
|
|
'app-name': text_type,
|
|
|
|
'app-version': text_type,
|
|
|
|
'branch': text_type,
|
|
|
|
'build-id': text_type,
|
|
|
|
'artifact-id': text_type,
|
|
|
|
'hash-type': text_type,
|
|
|
|
'platform': text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
},
|
|
|
|
|
|
|
|
Required('upstream-artifacts'): [{
|
|
|
|
Required('taskId'): taskref_or_string,
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('taskType'): text_type,
|
|
|
|
Required('paths'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
Required('zipExtract', default=False): bool,
|
|
|
|
}],
|
2018-11-28 11:13:52 +00:00
|
|
|
Optional('artifact-map'): object,
|
2018-10-26 19:02:36 +00:00
|
|
|
})
|
2018-08-16 16:23:15 +00:00
|
|
|
def build_beetmover_maven_payload(config, task, task_def):
|
|
|
|
build_beetmover_payload(config, task, task_def)
|
|
|
|
|
|
|
|
task_def['payload']['artifact_id'] = task['worker']['release-properties']['artifact-id']
|
2018-11-28 11:13:52 +00:00
|
|
|
if task['worker'].get('artifact-map'):
|
|
|
|
task_def['payload']['artifactMap'] = task['worker']['artifact-map']
|
2018-08-16 16:23:15 +00:00
|
|
|
|
|
|
|
del task_def['payload']['releaseProperties']['hashType']
|
|
|
|
del task_def['payload']['releaseProperties']['platform']
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('balrog', schema={
|
|
|
|
Required('balrog-action'): Any(*BALROG_ACTIONS),
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('product'): text_type,
|
|
|
|
Optional('platforms'): [text_type],
|
|
|
|
Optional('release-eta'): text_type,
|
|
|
|
Optional('channel-names'): optionally_keyed_by('release-type', [text_type]),
|
2018-10-26 19:02:36 +00:00
|
|
|
Optional('require-mirrors'): bool,
|
|
|
|
Optional('publish-rules'): optionally_keyed_by('release-type', 'release-level', [int]),
|
|
|
|
Optional('rules-to-update'): optionally_keyed_by(
|
2020-01-21 17:12:08 +00:00
|
|
|
'release-type', 'release-level', [text_type]),
|
|
|
|
Optional('archive-domain'): optionally_keyed_by('release-level', text_type),
|
|
|
|
Optional('download-domain'): optionally_keyed_by('release-level', text_type),
|
|
|
|
Optional('blob-suffix'): text_type,
|
|
|
|
Optional('complete-mar-filename-pattern'): text_type,
|
|
|
|
Optional('complete-mar-bouncer-product-pattern'): text_type,
|
2018-11-26 17:51:23 +00:00
|
|
|
Optional('update-line'): object,
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('suffixes'): [text_type],
|
2020-01-06 19:33:47 +00:00
|
|
|
Optional('background-rate'): optionally_keyed_by(
|
|
|
|
'release-type', 'beta-number', Any(int, None)),
|
|
|
|
Optional('force-fallback-mapping-update'): optionally_keyed_by(
|
|
|
|
'release-type', 'beta-number', bool),
|
2018-11-26 17:51:23 +00:00
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# list of artifact URLs for the artifacts that should be beetmoved
|
|
|
|
Optional('upstream-artifacts'): [{
|
|
|
|
# taskId of the task with the artifact
|
|
|
|
Required('taskId'): taskref_or_string,
|
|
|
|
|
|
|
|
# type of signing task (for CoT)
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('taskType'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# Paths to the artifacts to sign
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('paths'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
}],
|
|
|
|
})
|
2017-01-13 02:01:54 +00:00
|
|
|
def build_balrog_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
2018-02-16 02:14:26 +00:00
|
|
|
release_config = get_release_config(config)
|
2020-01-06 19:33:47 +00:00
|
|
|
beta_number = None
|
|
|
|
if 'b' in release_config['version']:
|
|
|
|
beta_number = release_config['version'].split('b')[-1]
|
2017-01-13 02:01:54 +00:00
|
|
|
|
2018-02-16 02:14:26 +00:00
|
|
|
if worker['balrog-action'] == 'submit-locale':
|
|
|
|
task_def['payload'] = {
|
2018-11-26 17:51:23 +00:00
|
|
|
'upstreamArtifacts': worker['upstream-artifacts'],
|
|
|
|
'suffixes': worker['suffixes'],
|
2018-02-16 02:14:26 +00:00
|
|
|
}
|
|
|
|
else:
|
|
|
|
for prop in ('archive-domain', 'channel-names', 'download-domain',
|
2020-01-06 19:33:47 +00:00
|
|
|
'publish-rules', 'rules-to-update', 'background-rate',
|
|
|
|
'force-fallback-mapping-update'):
|
2018-02-16 02:14:26 +00:00
|
|
|
if prop in worker:
|
|
|
|
resolve_keyed_by(
|
|
|
|
worker, prop, task['description'],
|
2018-09-11 21:09:28 +00:00
|
|
|
**{
|
2018-09-17 18:09:36 +00:00
|
|
|
'release-type': config.params['release_type'],
|
2018-09-11 21:09:28 +00:00
|
|
|
'release-level': config.params.release_level(),
|
2020-01-06 19:33:47 +00:00
|
|
|
'beta-number': beta_number,
|
2018-09-11 21:09:28 +00:00
|
|
|
}
|
2018-02-16 02:14:26 +00:00
|
|
|
)
|
|
|
|
task_def['payload'] = {
|
|
|
|
'build_number': release_config['build_number'],
|
|
|
|
'product': worker['product'],
|
|
|
|
'version': release_config['version'],
|
|
|
|
}
|
2018-08-27 19:21:30 +00:00
|
|
|
for prop in ('blob-suffix', 'complete-mar-filename-pattern',
|
|
|
|
'complete-mar-bouncer-product-pattern'):
|
|
|
|
if prop in worker:
|
|
|
|
task_def['payload'][prop.replace('-', '_')] = worker[prop]
|
2018-02-16 02:14:26 +00:00
|
|
|
if worker['balrog-action'] == 'submit-toplevel':
|
|
|
|
task_def['payload'].update({
|
|
|
|
'app_version': release_config['appVersion'],
|
|
|
|
'archive_domain': worker['archive-domain'],
|
|
|
|
'channel_names': worker['channel-names'],
|
|
|
|
'download_domain': worker['download-domain'],
|
|
|
|
'partial_versions': release_config.get('partial_versions', ""),
|
|
|
|
'platforms': worker['platforms'],
|
|
|
|
'rules_to_update': worker['rules-to-update'],
|
|
|
|
'require_mirrors': worker['require-mirrors'],
|
2018-11-26 17:51:23 +00:00
|
|
|
'update_line': worker['update-line'],
|
2018-02-16 02:14:26 +00:00
|
|
|
})
|
|
|
|
else: # schedule / ship
|
|
|
|
task_def['payload'].update({
|
|
|
|
'publish_rules': worker['publish-rules'],
|
2018-05-11 16:27:34 +00:00
|
|
|
'release_eta': worker.get('release-eta', config.params.get('release_eta')) or '',
|
2018-02-16 02:14:26 +00:00
|
|
|
})
|
2020-01-06 19:33:47 +00:00
|
|
|
if worker.get('force-fallback-mapping-update'):
|
|
|
|
task_def['payload']['force_fallback_mapping_update'] = \
|
|
|
|
worker['force-fallback-mapping-update']
|
|
|
|
if worker.get('background-rate'):
|
|
|
|
task_def['payload']['background_rate'] = worker['background-rate']
|
2017-01-13 02:01:54 +00:00
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('bouncer-aliases', schema={
|
|
|
|
Required('entries'): object,
|
|
|
|
})
|
2018-02-26 14:21:54 +00:00
|
|
|
def build_bouncer_aliases_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'aliases_entries': worker['entries']
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('bouncer-locations', schema={
|
|
|
|
Required('implementation'): 'bouncer-locations',
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('bouncer-products'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
})
|
2018-09-05 09:34:26 +00:00
|
|
|
def build_bouncer_locations_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
|
|
|
release_config = get_release_config(config)
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'bouncer_products': worker['bouncer-products'],
|
|
|
|
'version': release_config['version'],
|
2019-07-26 13:25:35 +00:00
|
|
|
'product': task['shipping-product'],
|
2018-09-05 09:34:26 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('bouncer-submission', schema={
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('locales'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
Required('entries'): object,
|
|
|
|
})
|
2018-02-26 14:14:46 +00:00
|
|
|
def build_bouncer_submission_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'locales': worker['locales'],
|
|
|
|
'submission_entries': worker['entries']
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('push-apk', schema={
|
|
|
|
Required('upstream-artifacts'): [{
|
|
|
|
Required('taskId'): taskref_or_string,
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('taskType'): text_type,
|
|
|
|
Required('paths'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
Optional('optional', default=False): bool,
|
|
|
|
}],
|
|
|
|
|
|
|
|
# "Invalid" is a noop for try and other non-supported branches
|
|
|
|
Required('google-play-track'): Any('production', 'beta', 'alpha', 'rollout', 'internal'),
|
|
|
|
Required('commit'): bool,
|
|
|
|
Optional('rollout-percentage'): Any(int, None),
|
|
|
|
})
|
2017-04-04 09:21:07 +00:00
|
|
|
def build_push_apk_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
2017-11-28 12:11:07 +00:00
|
|
|
'commit': worker['commit'],
|
2018-03-30 16:22:38 +00:00
|
|
|
'upstreamArtifacts': worker['upstream-artifacts'],
|
2017-04-04 09:21:07 +00:00
|
|
|
'google_play_track': worker['google-play-track'],
|
|
|
|
}
|
|
|
|
|
2017-04-07 08:49:32 +00:00
|
|
|
if worker.get('rollout-percentage', None):
|
|
|
|
task_def['payload']['rollout_percentage'] = worker['rollout-percentage']
|
|
|
|
|
2017-04-04 09:21:07 +00:00
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('push-snap', schema={
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('channel'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
Required('upstream-artifacts'): [{
|
|
|
|
Required('taskId'): taskref_or_string,
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('taskType'): text_type,
|
|
|
|
Required('paths'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
}],
|
|
|
|
})
|
2018-03-20 16:19:50 +00:00
|
|
|
def build_push_snap_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
2019-06-30 21:10:14 +00:00
|
|
|
'channel': worker['channel'],
|
2018-03-20 16:19:50 +00:00
|
|
|
'upstreamArtifacts': worker['upstream-artifacts'],
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2020-03-09 12:16:51 +00:00
|
|
|
@payload_builder('push-flatpak', schema={
|
|
|
|
Required('channel'): text_type,
|
|
|
|
Required('upstream-artifacts'): [{
|
|
|
|
Required('taskId'): taskref_or_string,
|
|
|
|
Required('taskType'): text_type,
|
|
|
|
Required('paths'): [text_type],
|
|
|
|
}],
|
|
|
|
})
|
|
|
|
def build_push_flatpak_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'channel': worker['channel'],
|
|
|
|
'upstreamArtifacts': worker['upstream-artifacts'],
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('shipit-shipped', schema={
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('release-name'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
})
|
2018-06-02 00:41:22 +00:00
|
|
|
def build_ship_it_shipped_payload(config, task, task_def):
|
2018-01-25 16:07:07 +00:00
|
|
|
worker = task['worker']
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'release_name': worker['release-name']
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2019-11-29 15:12:08 +00:00
|
|
|
@payload_builder('shipit-maybe-release', schema={
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('phase'): text_type,
|
2019-11-29 15:12:08 +00:00
|
|
|
})
|
|
|
|
def build_ship_it_maybe_release_payload(config, task, task_def):
|
|
|
|
# expect branch name, including path
|
|
|
|
branch = config.params['head_repository'][len('https://hg.mozilla.org/'):]
|
|
|
|
# 'version' is e.g. '71.0b13' (app_version doesn't have beta number)
|
|
|
|
version = config.params['version']
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'product': task['shipping-product'],
|
|
|
|
'branch': branch,
|
|
|
|
'phase': task['worker']['phase'],
|
|
|
|
'version': version,
|
|
|
|
'cron_revision': config.params['head_rev'],
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2019-08-09 18:53:50 +00:00
|
|
|
@payload_builder('push-addons', schema={
|
2018-10-26 19:02:36 +00:00
|
|
|
Required('channel'): Any('listed', 'unlisted'),
|
|
|
|
Required('upstream-artifacts'): [{
|
|
|
|
Required('taskId'): taskref_or_string,
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('taskType'): text_type,
|
|
|
|
Required('paths'): [text_type],
|
2018-10-26 19:02:36 +00:00
|
|
|
}],
|
|
|
|
})
|
2019-08-09 18:53:50 +00:00
|
|
|
def build_push_addons_payload(config, task, task_def):
|
2018-03-30 16:22:38 +00:00
|
|
|
worker = task['worker']
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'channel': worker['channel'],
|
|
|
|
'upstreamArtifacts': worker['upstream-artifacts'],
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('treescript', schema={
|
|
|
|
Required('tags'): [Any('buildN', 'release', None)],
|
|
|
|
Required('bump'): bool,
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('bump-files'): [text_type],
|
|
|
|
Optional('repo-param-prefix'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
Optional('dontbuild'): bool,
|
2019-11-22 23:57:25 +00:00
|
|
|
Optional('ignore-closed-tree'): bool,
|
2018-10-26 19:02:36 +00:00
|
|
|
Required('force-dry-run', default=True): bool,
|
2019-11-22 23:57:25 +00:00
|
|
|
Required('push', default=False): bool,
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('source-repo'): text_type,
|
2020-02-25 10:48:54 +00:00
|
|
|
Optional('ssh-user'): text_type,
|
2019-11-22 23:57:25 +00:00
|
|
|
Optional('l10n-bump-info'): {
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('name'): text_type,
|
|
|
|
Required('path'): text_type,
|
|
|
|
Required('version-path'): text_type,
|
|
|
|
Optional('revision-url'): text_type,
|
2019-11-22 23:57:25 +00:00
|
|
|
Optional('ignore-config'): object,
|
|
|
|
Required('platform-configs'): [{
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('platforms'): [text_type],
|
|
|
|
Required('path'): text_type,
|
|
|
|
Optional('format'): text_type,
|
2019-11-22 23:57:25 +00:00
|
|
|
}],
|
|
|
|
},
|
2020-02-25 10:48:54 +00:00
|
|
|
Optional('merge-info'): object,
|
2018-10-26 19:02:36 +00:00
|
|
|
})
|
2018-02-21 12:06:38 +00:00
|
|
|
def build_treescript_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
|
|
|
release_config = get_release_config(config)
|
|
|
|
|
2019-11-22 23:57:25 +00:00
|
|
|
task_def['payload'] = {'actions': []}
|
|
|
|
actions = task_def['payload']['actions']
|
2018-06-21 17:56:07 +00:00
|
|
|
if worker['tags']:
|
|
|
|
tag_names = []
|
2018-02-21 12:06:38 +00:00
|
|
|
product = task['shipping-product'].upper()
|
|
|
|
version = release_config['version'].replace('.', '_')
|
|
|
|
buildnum = release_config['build_number']
|
2018-06-21 17:56:07 +00:00
|
|
|
if 'buildN' in worker['tags']:
|
|
|
|
tag_names.extend([
|
|
|
|
"{}_{}_BUILD{}".format(product, version, buildnum),
|
|
|
|
])
|
|
|
|
if 'release' in worker['tags']:
|
|
|
|
tag_names.extend([
|
|
|
|
"{}_{}_RELEASE".format(product, version)
|
|
|
|
])
|
2018-02-21 12:06:38 +00:00
|
|
|
tag_info = {
|
|
|
|
'tags': tag_names,
|
2018-05-23 20:56:25 +00:00
|
|
|
'revision': config.params['{}head_rev'.format(worker.get('repo-param-prefix', ''))],
|
2018-02-21 12:06:38 +00:00
|
|
|
}
|
|
|
|
task_def['payload']['tag_info'] = tag_info
|
2019-11-22 23:57:25 +00:00
|
|
|
actions.append('tag')
|
2018-02-21 12:06:38 +00:00
|
|
|
|
|
|
|
if worker['bump']:
|
|
|
|
if not worker['bump-files']:
|
|
|
|
raise Exception("Version Bump requested without bump-files")
|
|
|
|
|
|
|
|
bump_info = {}
|
|
|
|
bump_info['next_version'] = release_config['next_version']
|
|
|
|
bump_info['files'] = worker['bump-files']
|
|
|
|
task_def['payload']['version_bump_info'] = bump_info
|
2019-11-22 23:57:25 +00:00
|
|
|
actions.append('version_bump')
|
|
|
|
|
|
|
|
if worker.get('l10n-bump-info'):
|
|
|
|
l10n_bump_info = {}
|
|
|
|
for k, v in worker['l10n-bump-info'].items():
|
|
|
|
l10n_bump_info[k.replace('-', '_')] = worker['l10n-bump-info'][k]
|
|
|
|
task_def['payload']['l10n_bump_info'] = [l10n_bump_info]
|
|
|
|
actions.append('l10n_bump')
|
2018-02-21 12:06:38 +00:00
|
|
|
|
2020-02-25 10:48:54 +00:00
|
|
|
if worker.get('merge-info'):
|
|
|
|
merge_info = {}
|
|
|
|
for k, v in worker['merge-info'].items():
|
|
|
|
merge_info[k.replace('-', '_')] = worker['merge-info'][k]
|
|
|
|
task_def['payload']['merge_info'] = merge_info
|
|
|
|
actions.append('merge_day')
|
|
|
|
|
2018-02-21 12:06:38 +00:00
|
|
|
if worker['push']:
|
2019-11-22 23:57:25 +00:00
|
|
|
actions.append('push')
|
2018-02-21 12:06:38 +00:00
|
|
|
|
|
|
|
if worker.get('force-dry-run'):
|
|
|
|
task_def['payload']['dry_run'] = True
|
|
|
|
|
2018-07-27 20:42:59 +00:00
|
|
|
if worker.get('dontbuild'):
|
2018-09-01 02:17:45 +00:00
|
|
|
task_def['payload']['dontbuild'] = True
|
2018-07-27 20:42:59 +00:00
|
|
|
|
2019-11-22 23:57:25 +00:00
|
|
|
if worker.get('ignore-closed-tree') is not None:
|
|
|
|
task_def['payload']['ignore_closed_tree'] = worker['ignore-closed-tree']
|
|
|
|
|
|
|
|
if worker.get('source-repo'):
|
|
|
|
task_def['payload']['source_repo'] = worker['source-repo']
|
|
|
|
|
2020-02-25 10:48:54 +00:00
|
|
|
if worker.get('ssh-user'):
|
|
|
|
task_def['payload']['ssh_user'] = worker['ssh-user']
|
|
|
|
|
2018-02-21 12:06:38 +00:00
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('invalid', schema={
|
|
|
|
# an invalid task is one which should never actually be created; this is used in
|
|
|
|
# release automation on branches where the task just doesn't make sense
|
|
|
|
Extra: object,
|
|
|
|
})
|
Bug 1359976: base worker payload generation on worker-type; r=wcosta r=aki
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
--HG--
extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
2017-05-08 22:53:50 +00:00
|
|
|
def build_invalid_payload(config, task, task_def):
|
|
|
|
task_def['payload'] = 'invalid task - should never be created'
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('always-optimized', schema={
|
|
|
|
Extra: object,
|
|
|
|
})
|
2019-07-02 17:05:23 +00:00
|
|
|
@payload_builder('succeed', schema={
|
|
|
|
})
|
|
|
|
def build_dummy_payload(config, task, task_def):
|
2017-12-28 03:14:34 +00:00
|
|
|
task_def['payload'] = {}
|
|
|
|
|
|
|
|
|
2018-10-26 19:02:36 +00:00
|
|
|
@payload_builder('script-engine-autophone', schema={
|
|
|
|
Required('os'): Any('macosx', 'linux'),
|
|
|
|
|
|
|
|
# A link for an executable to download
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('context'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# Tells the worker whether machine should reboot
|
|
|
|
# after the task is finished.
|
|
|
|
Optional('reboot'):
|
|
|
|
Any(False, 'always', 'never', 'on-exception', 'on-failure'),
|
|
|
|
|
|
|
|
# the command to run
|
|
|
|
Optional('command'): [taskref_or_string],
|
|
|
|
|
|
|
|
# environment variables
|
2020-01-21 17:12:08 +00:00
|
|
|
Optional('env'): {text_type: taskref_or_string},
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# artifacts to extract from the task image after completion
|
|
|
|
Optional('artifacts'): [{
|
|
|
|
# type of artifact -- simple file, or recursive directory
|
|
|
|
Required('type'): Any('file', 'directory'),
|
|
|
|
|
|
|
|
# task image path from which to read artifact
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('path'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
|
|
|
|
# name of the produced artifact (root of the names for
|
|
|
|
# type=directory)
|
2020-01-21 17:12:08 +00:00
|
|
|
Required('name'): text_type,
|
2018-10-26 19:02:36 +00:00
|
|
|
}],
|
|
|
|
})
|
2018-06-22 23:39:32 +00:00
|
|
|
def build_script_engine_autophone_payload(config, task, task_def):
|
|
|
|
worker = task['worker']
|
|
|
|
artifacts = map(lambda artifact: {
|
|
|
|
'name': artifact['name'],
|
|
|
|
'path': artifact['path'],
|
|
|
|
'type': artifact['type'],
|
|
|
|
'expires': task_def['expires'],
|
|
|
|
}, worker.get('artifacts', []))
|
|
|
|
|
|
|
|
task_def['payload'] = {
|
|
|
|
'context': worker['context'],
|
|
|
|
'command': worker['command'],
|
|
|
|
'env': worker['env'],
|
|
|
|
'artifacts': artifacts,
|
|
|
|
}
|
|
|
|
if worker.get('reboot'):
|
|
|
|
task_def['payload'] = worker['reboot']
|
|
|
|
|
|
|
|
if task.get('needs-sccache'):
|
|
|
|
raise Exception('needs-sccache not supported in taskcluster-worker')
|
|
|
|
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
transforms = TransformSequence()
|
|
|
|
|
|
|
|
|
2019-07-12 20:44:12 +00:00
|
|
|
@transforms.add
|
|
|
|
def set_implementation(config, tasks):
|
|
|
|
"""
|
|
|
|
Set the worker implementation based on the worker-type alias.
|
|
|
|
"""
|
|
|
|
for task in tasks:
|
|
|
|
if 'implementation' in task['worker']:
|
|
|
|
yield task
|
|
|
|
continue
|
|
|
|
|
|
|
|
impl, os = worker_type_implementation(config.graph_config, task['worker-type'])
|
|
|
|
|
|
|
|
tags = task.setdefault('tags', {})
|
|
|
|
tags['worker-implementation'] = impl
|
|
|
|
if os:
|
|
|
|
task['tags']['os'] = os
|
|
|
|
worker = task.setdefault('worker', {})
|
|
|
|
worker['implementation'] = impl
|
|
|
|
if os:
|
|
|
|
worker['os'] = os
|
|
|
|
|
|
|
|
yield task
|
|
|
|
|
|
|
|
|
2017-12-08 03:23:57 +00:00
|
|
|
@transforms.add
|
|
|
|
def set_defaults(config, tasks):
|
|
|
|
for task in tasks:
|
|
|
|
task.setdefault('shipping-phase', None)
|
|
|
|
task.setdefault('shipping-product', None)
|
|
|
|
task.setdefault('always-target', False)
|
|
|
|
task.setdefault('optimization', None)
|
|
|
|
task.setdefault('needs-sccache', False)
|
|
|
|
|
|
|
|
worker = task['worker']
|
2019-02-12 21:12:27 +00:00
|
|
|
if worker['implementation'] in ('docker-worker',):
|
2017-12-08 03:23:57 +00:00
|
|
|
worker.setdefault('chain-of-trust', False)
|
|
|
|
worker.setdefault('taskcluster-proxy', False)
|
2020-03-17 19:06:34 +00:00
|
|
|
worker.setdefault('allow-ptrace', True)
|
2017-12-08 03:23:57 +00:00
|
|
|
worker.setdefault('loopback-video', False)
|
|
|
|
worker.setdefault('loopback-audio', False)
|
|
|
|
worker.setdefault('docker-in-docker', False)
|
2018-08-07 16:05:38 +00:00
|
|
|
worker.setdefault('privileged', False)
|
2017-12-08 03:23:57 +00:00
|
|
|
worker.setdefault('volumes', [])
|
|
|
|
worker.setdefault('env', {})
|
|
|
|
if 'caches' in worker:
|
|
|
|
for c in worker['caches']:
|
|
|
|
c.setdefault('skip-untrusted', False)
|
|
|
|
elif worker['implementation'] == 'generic-worker':
|
|
|
|
worker.setdefault('env', {})
|
|
|
|
worker.setdefault('os-groups', [])
|
2018-08-29 17:13:40 +00:00
|
|
|
if worker['os-groups'] and worker['os'] != 'windows':
|
|
|
|
raise Exception('os-groups feature of generic-worker is only supported on '
|
|
|
|
'Windows, not on {}'.format(worker['os']))
|
2017-12-08 03:23:57 +00:00
|
|
|
worker.setdefault('chain-of-trust', False)
|
2018-08-16 16:23:15 +00:00
|
|
|
elif worker['implementation'] in (
|
|
|
|
'scriptworker-signing', 'beetmover', 'beetmover-push-to-release', 'beetmover-maven',
|
|
|
|
):
|
2017-12-08 03:23:57 +00:00
|
|
|
worker.setdefault('max-run-time', 600)
|
|
|
|
elif worker['implementation'] == 'push-apk':
|
|
|
|
worker.setdefault('commit', False)
|
|
|
|
|
|
|
|
yield task
|
|
|
|
|
|
|
|
|
2017-10-30 00:35:03 +00:00
|
|
|
@transforms.add
|
|
|
|
def task_name_from_label(config, tasks):
|
|
|
|
for task in tasks:
|
|
|
|
if 'label' not in task:
|
|
|
|
if 'name' not in task:
|
|
|
|
raise Exception("task has neither a name nor a label")
|
|
|
|
task['label'] = '{}-{}'.format(config.kind, task['name'])
|
|
|
|
if task.get('name'):
|
|
|
|
del task['name']
|
|
|
|
yield task
|
|
|
|
|
|
|
|
|
2018-04-19 15:21:30 +00:00
|
|
|
UNSUPPORTED_SHIPPING_PRODUCT_ERROR = """\
|
|
|
|
The shipping product {product} is not in the list of configured products in
|
|
|
|
`taskcluster/ci/config.yml'.
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
def validate_shipping_product(config, product):
|
|
|
|
if product not in config.graph_config['release-promotion']['products']:
|
|
|
|
raise Exception(UNSUPPORTED_SHIPPING_PRODUCT_ERROR.format(product=product))
|
|
|
|
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
@transforms.add
|
|
|
|
def validate(config, tasks):
|
|
|
|
for task in tasks:
|
2017-12-08 03:23:57 +00:00
|
|
|
validate_schema(
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
task_description_schema, task,
|
|
|
|
"In task {!r}:".format(task.get('label', '?no-label?')))
|
2018-10-26 19:02:36 +00:00
|
|
|
validate_schema(
|
|
|
|
payload_builders[task['worker']['implementation']].schema,
|
|
|
|
task['worker'],
|
|
|
|
"In task.run {!r}:".format(task.get('label', '?no-label?')))
|
2018-04-19 15:21:30 +00:00
|
|
|
if task['shipping-product'] is not None:
|
|
|
|
validate_shipping_product(config, task['shipping-product'])
|
2017-12-08 03:23:57 +00:00
|
|
|
yield task
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
|
|
|
|
2017-01-09 21:23:04 +00:00
|
|
|
@index_builder('generic')
|
|
|
|
def add_generic_index_routes(config, task):
|
|
|
|
index = task.get('index')
|
|
|
|
routes = task.setdefault('routes', [])
|
|
|
|
|
2017-12-13 23:00:14 +00:00
|
|
|
verify_index(config, index)
|
2017-01-09 21:23:04 +00:00
|
|
|
|
|
|
|
subs = config.params.copy()
|
2017-11-08 04:57:35 +00:00
|
|
|
subs['job-name'] = index['job-name']
|
2017-01-09 21:23:04 +00:00
|
|
|
subs['build_date_long'] = time.strftime("%Y.%m.%d.%Y%m%d%H%M%S",
|
|
|
|
time.gmtime(config.params['build_date']))
|
2019-09-05 21:11:45 +00:00
|
|
|
subs['build_date'] = time.strftime("%Y.%m.%d",
|
|
|
|
time.gmtime(config.params['build_date']))
|
2017-01-09 21:23:04 +00:00
|
|
|
subs['product'] = index['product']
|
2017-12-04 22:47:37 +00:00
|
|
|
subs['trust-domain'] = config.graph_config['trust-domain']
|
2018-03-07 22:23:00 +00:00
|
|
|
subs['branch_rev'] = get_branch_rev(config)
|
2017-01-09 21:23:04 +00:00
|
|
|
|
2017-07-20 01:24:12 +00:00
|
|
|
project = config.params.get('project')
|
|
|
|
|
2017-01-09 21:23:04 +00:00
|
|
|
for tpl in V2_ROUTE_TEMPLATES:
|
|
|
|
routes.append(tpl.format(**subs))
|
|
|
|
|
2017-07-20 01:24:12 +00:00
|
|
|
# Additionally alias all tasks for "trunk" repos into a common
|
|
|
|
# namespace.
|
|
|
|
if project and project in TRUNK_PROJECTS:
|
|
|
|
for tpl in V2_TRUNK_ROUTE_TEMPLATES:
|
|
|
|
routes.append(tpl.format(**subs))
|
|
|
|
|
2017-01-09 21:23:04 +00:00
|
|
|
return task
|
|
|
|
|
|
|
|
|
|
|
|
@index_builder('nightly')
|
|
|
|
def add_nightly_index_routes(config, task):
|
|
|
|
index = task.get('index')
|
|
|
|
routes = task.setdefault('routes', [])
|
|
|
|
|
2017-12-13 23:00:14 +00:00
|
|
|
verify_index(config, index)
|
2017-01-09 21:23:04 +00:00
|
|
|
|
|
|
|
subs = config.params.copy()
|
2017-11-08 04:57:35 +00:00
|
|
|
subs['job-name'] = index['job-name']
|
2017-01-09 21:23:04 +00:00
|
|
|
subs['build_date_long'] = time.strftime("%Y.%m.%d.%Y%m%d%H%M%S",
|
|
|
|
time.gmtime(config.params['build_date']))
|
|
|
|
subs['build_date'] = time.strftime("%Y.%m.%d",
|
|
|
|
time.gmtime(config.params['build_date']))
|
|
|
|
subs['product'] = index['product']
|
2017-12-04 22:47:37 +00:00
|
|
|
subs['trust-domain'] = config.graph_config['trust-domain']
|
2018-03-07 22:23:00 +00:00
|
|
|
subs['branch_rev'] = get_branch_rev(config)
|
2017-01-09 21:23:04 +00:00
|
|
|
|
|
|
|
for tpl in V2_NIGHTLY_TEMPLATES:
|
|
|
|
routes.append(tpl.format(**subs))
|
|
|
|
|
2017-01-19 19:25:35 +00:00
|
|
|
# Also add routes for en-US
|
|
|
|
task = add_l10n_index_routes(config, task, force_locale="en-US")
|
|
|
|
|
|
|
|
return task
|
|
|
|
|
|
|
|
|
2019-03-22 03:22:49 +00:00
|
|
|
@index_builder('shippable')
|
|
|
|
def add_shippable_index_routes(config, task):
|
2019-03-22 03:24:44 +00:00
|
|
|
index = task.get('index')
|
|
|
|
routes = task.setdefault('routes', [])
|
|
|
|
|
|
|
|
verify_index(config, index)
|
|
|
|
|
|
|
|
subs = config.params.copy()
|
|
|
|
subs['job-name'] = index['job-name']
|
|
|
|
subs['build_date_long'] = time.strftime("%Y.%m.%d.%Y%m%d%H%M%S",
|
|
|
|
time.gmtime(config.params['build_date']))
|
|
|
|
subs['build_date'] = time.strftime("%Y.%m.%d",
|
|
|
|
time.gmtime(config.params['build_date']))
|
|
|
|
subs['product'] = index['product']
|
|
|
|
subs['trust-domain'] = config.graph_config['trust-domain']
|
|
|
|
subs['branch_rev'] = get_branch_rev(config)
|
|
|
|
|
|
|
|
for tpl in V2_SHIPPABLE_TEMPLATES:
|
|
|
|
routes.append(tpl.format(**subs))
|
|
|
|
|
|
|
|
# Also add routes for en-US
|
|
|
|
task = add_shippable_l10n_index_routes(config, task, force_locale="en-US")
|
|
|
|
|
|
|
|
# For nightly-compat index:
|
2019-03-27 13:45:44 +00:00
|
|
|
if 'nightly' in config.params['target_tasks_method']:
|
|
|
|
add_nightly_index_routes(config, task)
|
2019-03-22 03:24:44 +00:00
|
|
|
|
2019-03-22 03:22:49 +00:00
|
|
|
return task
|
|
|
|
|
|
|
|
|
2017-01-19 19:25:35 +00:00
|
|
|
@index_builder('nightly-with-multi-l10n')
|
|
|
|
def add_nightly_multi_index_routes(config, task):
|
|
|
|
task = add_nightly_index_routes(config, task)
|
|
|
|
task = add_l10n_index_routes(config, task, force_locale="multi")
|
2017-01-09 21:23:04 +00:00
|
|
|
return task
|
|
|
|
|
|
|
|
|
|
|
|
@index_builder('l10n')
|
2017-01-19 19:25:35 +00:00
|
|
|
def add_l10n_index_routes(config, task, force_locale=None):
|
2017-01-09 21:23:04 +00:00
|
|
|
index = task.get('index')
|
|
|
|
routes = task.setdefault('routes', [])
|
|
|
|
|
2017-12-13 23:00:14 +00:00
|
|
|
verify_index(config, index)
|
2017-01-09 21:23:04 +00:00
|
|
|
|
|
|
|
subs = config.params.copy()
|
2017-11-08 04:57:35 +00:00
|
|
|
subs['job-name'] = index['job-name']
|
2017-01-09 21:23:04 +00:00
|
|
|
subs['build_date_long'] = time.strftime("%Y.%m.%d.%Y%m%d%H%M%S",
|
|
|
|
time.gmtime(config.params['build_date']))
|
|
|
|
subs['product'] = index['product']
|
2017-12-04 22:47:37 +00:00
|
|
|
subs['trust-domain'] = config.graph_config['trust-domain']
|
2018-03-07 22:23:00 +00:00
|
|
|
subs['branch_rev'] = get_branch_rev(config)
|
2017-01-09 21:23:04 +00:00
|
|
|
|
|
|
|
locales = task['attributes'].get('chunk_locales',
|
|
|
|
task['attributes'].get('all_locales'))
|
2017-07-07 17:58:06 +00:00
|
|
|
# Some tasks has only one locale set
|
|
|
|
if task['attributes'].get('locale'):
|
|
|
|
locales = [task['attributes']['locale']]
|
2017-01-09 21:23:04 +00:00
|
|
|
|
2017-01-19 19:25:35 +00:00
|
|
|
if force_locale:
|
|
|
|
# Used for en-US and multi-locale
|
|
|
|
locales = [force_locale]
|
|
|
|
|
2017-01-09 21:23:04 +00:00
|
|
|
if not locales:
|
|
|
|
raise Exception("Error: Unable to use l10n index for tasks without locales")
|
|
|
|
|
|
|
|
# If there are too many locales, we can't write a route for all of them
|
|
|
|
# See Bug 1323792
|
|
|
|
if len(locales) > 18: # 18 * 3 = 54, max routes = 64
|
|
|
|
return task
|
|
|
|
|
|
|
|
for locale in locales:
|
|
|
|
for tpl in V2_L10N_TEMPLATES:
|
|
|
|
routes.append(tpl.format(locale=locale, **subs))
|
|
|
|
|
|
|
|
return task
|
|
|
|
|
|
|
|
|
2019-03-22 03:24:44 +00:00
|
|
|
@index_builder('shippable-l10n')
|
|
|
|
def add_shippable_l10n_index_routes(config, task, force_locale=None):
|
|
|
|
index = task.get('index')
|
|
|
|
routes = task.setdefault('routes', [])
|
|
|
|
|
|
|
|
verify_index(config, index)
|
|
|
|
|
|
|
|
subs = config.params.copy()
|
|
|
|
subs['job-name'] = index['job-name']
|
|
|
|
subs['build_date_long'] = time.strftime("%Y.%m.%d.%Y%m%d%H%M%S",
|
|
|
|
time.gmtime(config.params['build_date']))
|
|
|
|
subs['product'] = index['product']
|
|
|
|
subs['trust-domain'] = config.graph_config['trust-domain']
|
|
|
|
subs['branch_rev'] = get_branch_rev(config)
|
|
|
|
|
|
|
|
locales = task['attributes'].get('chunk_locales',
|
|
|
|
task['attributes'].get('all_locales'))
|
|
|
|
# Some tasks has only one locale set
|
|
|
|
if task['attributes'].get('locale'):
|
|
|
|
locales = [task['attributes']['locale']]
|
|
|
|
|
|
|
|
if force_locale:
|
|
|
|
# Used for en-US and multi-locale
|
|
|
|
locales = [force_locale]
|
|
|
|
|
|
|
|
if not locales:
|
|
|
|
raise Exception("Error: Unable to use l10n index for tasks without locales")
|
|
|
|
|
|
|
|
# If there are too many locales, we can't write a route for all of them
|
|
|
|
# See Bug 1323792
|
|
|
|
if len(locales) > 18: # 18 * 3 = 54, max routes = 64
|
|
|
|
return task
|
|
|
|
|
|
|
|
for locale in locales:
|
|
|
|
for tpl in V2_SHIPPABLE_L10N_TEMPLATES:
|
|
|
|
routes.append(tpl.format(locale=locale, **subs))
|
|
|
|
|
2019-03-27 13:45:44 +00:00
|
|
|
# For nightly-compat index:
|
|
|
|
if 'nightly' in config.params['target_tasks_method']:
|
|
|
|
add_nightly_l10n_index_routes(config, task, force_locale)
|
|
|
|
|
2019-03-22 03:24:44 +00:00
|
|
|
return task
|
|
|
|
|
|
|
|
|
2017-01-12 16:17:32 +00:00
|
|
|
@index_builder('nightly-l10n')
|
|
|
|
def add_nightly_l10n_index_routes(config, task, force_locale=None):
|
|
|
|
index = task.get('index')
|
|
|
|
routes = task.setdefault('routes', [])
|
|
|
|
|
2017-12-13 23:00:14 +00:00
|
|
|
verify_index(config, index)
|
2017-01-12 16:17:32 +00:00
|
|
|
|
|
|
|
subs = config.params.copy()
|
|
|
|
subs['job-name'] = index['job-name']
|
|
|
|
subs['build_date_long'] = time.strftime("%Y.%m.%d.%Y%m%d%H%M%S",
|
|
|
|
time.gmtime(config.params['build_date']))
|
2017-12-01 16:03:01 +00:00
|
|
|
subs['build_date'] = time.strftime("%Y.%m.%d",
|
|
|
|
time.gmtime(config.params['build_date']))
|
2017-01-12 16:17:32 +00:00
|
|
|
subs['product'] = index['product']
|
2017-12-04 22:47:37 +00:00
|
|
|
subs['trust-domain'] = config.graph_config['trust-domain']
|
2018-03-07 22:23:00 +00:00
|
|
|
subs['branch_rev'] = get_branch_rev(config)
|
2017-01-12 16:17:32 +00:00
|
|
|
|
|
|
|
locales = task['attributes'].get('chunk_locales',
|
|
|
|
task['attributes'].get('all_locales'))
|
|
|
|
# Some tasks has only one locale set
|
|
|
|
if task['attributes'].get('locale'):
|
|
|
|
locales = [task['attributes']['locale']]
|
|
|
|
|
|
|
|
if force_locale:
|
|
|
|
# Used for en-US and multi-locale
|
|
|
|
locales = [force_locale]
|
|
|
|
|
|
|
|
if not locales:
|
|
|
|
raise Exception("Error: Unable to use l10n index for tasks without locales")
|
|
|
|
|
|
|
|
for locale in locales:
|
|
|
|
for tpl in V2_NIGHTLY_L10N_TEMPLATES:
|
|
|
|
routes.append(tpl.format(locale=locale, **subs))
|
|
|
|
|
|
|
|
return task
|
|
|
|
|
|
|
|
|
2019-04-16 09:46:07 +00:00
|
|
|
def add_geckoview_index_routes(config, task):
|
|
|
|
index = task.get('index')
|
|
|
|
routes = task.setdefault('routes', [])
|
|
|
|
geckoview_version = _compute_geckoview_version(
|
|
|
|
config.params['app_version'],
|
|
|
|
config.params['moz_build_date']
|
|
|
|
)
|
|
|
|
|
|
|
|
subs = {
|
|
|
|
'geckoview-version': geckoview_version,
|
|
|
|
'job-name': index['job-name'],
|
|
|
|
'product': index['product'],
|
|
|
|
'project': config.params['project'],
|
|
|
|
'trust-domain': config.graph_config['trust-domain'],
|
|
|
|
}
|
|
|
|
routes.append(V2_GECKOVIEW_RELEASE.format(**subs))
|
|
|
|
|
|
|
|
return task
|
|
|
|
|
|
|
|
|
|
|
|
@index_builder('android-nightly')
|
|
|
|
def add_android_nightly_index_routes(config, task):
|
|
|
|
task = add_nightly_index_routes(config, task)
|
|
|
|
task = add_geckoview_index_routes(config, task)
|
|
|
|
|
|
|
|
return task
|
|
|
|
|
|
|
|
|
|
|
|
@index_builder('android-nightly-with-multi-l10n')
|
|
|
|
def add_android_nightly_multi_index_routes(config, task):
|
|
|
|
task = add_nightly_multi_index_routes(config, task)
|
|
|
|
task = add_geckoview_index_routes(config, task)
|
|
|
|
|
|
|
|
return task
|
|
|
|
|
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
@transforms.add
|
|
|
|
def add_index_routes(config, tasks):
|
|
|
|
for task in tasks:
|
2017-12-04 22:58:20 +00:00
|
|
|
index = task.get('index', {})
|
2016-09-12 15:51:49 +00:00
|
|
|
|
2016-09-28 11:20:41 +00:00
|
|
|
# The default behavior is to rank tasks according to their tier
|
|
|
|
extra_index = task.setdefault('extra', {}).setdefault('index', {})
|
|
|
|
rank = index.get('rank', 'by-tier')
|
|
|
|
|
|
|
|
if rank == 'by-tier':
|
|
|
|
# rank is zero for non-tier-1 tasks and based on pushid for others;
|
|
|
|
# this sorts tier-{2,3} builds below tier-1 in the index
|
|
|
|
tier = task.get('treeherder', {}).get('tier', 3)
|
2016-11-03 02:23:27 +00:00
|
|
|
extra_index['rank'] = 0 if tier > 1 else int(config.params['build_date'])
|
|
|
|
elif rank == 'build_date':
|
|
|
|
extra_index['rank'] = int(config.params['build_date'])
|
2016-09-28 11:20:41 +00:00
|
|
|
else:
|
|
|
|
extra_index['rank'] = rank
|
|
|
|
|
2017-12-04 22:58:20 +00:00
|
|
|
if not index:
|
|
|
|
yield task
|
|
|
|
continue
|
|
|
|
|
|
|
|
index_type = index.get('type', 'generic')
|
|
|
|
task = index_builders[index_type](config, task)
|
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
del task['index']
|
|
|
|
yield task
|
|
|
|
|
|
|
|
|
2019-11-22 23:28:57 +00:00
|
|
|
@transforms.add
|
|
|
|
def try_task_config_env(config, tasks):
|
|
|
|
"""Set environment variables in the task."""
|
|
|
|
env = config.params['try_task_config'].get('env')
|
|
|
|
# Find all implementations that have an 'env' key.
|
|
|
|
implementations = {name for name, builder in payload_builders.items()
|
|
|
|
if 'env' in builder.schema.schema}
|
|
|
|
for task in tasks:
|
|
|
|
if env and task['worker']['implementation'] in implementations:
|
|
|
|
task['worker']['env'].update(env)
|
|
|
|
yield task
|
|
|
|
|
|
|
|
|
|
|
|
@transforms.add
|
|
|
|
def try_task_config_chemspill_prio(config, tasks):
|
|
|
|
"""Increase the priority from lowest and very-low -> low, but leave others unchanged."""
|
|
|
|
chemspill_prio = config.params['try_task_config'].get('chemspill-prio')
|
|
|
|
for task in tasks:
|
|
|
|
if chemspill_prio and task['priority'] in ('lowest', 'very-low'):
|
|
|
|
task['priority'] = 'low'
|
|
|
|
yield task
|
|
|
|
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
@transforms.add
|
|
|
|
def build_task(config, tasks):
|
|
|
|
for task in tasks:
|
2017-08-08 19:13:05 +00:00
|
|
|
level = str(config.params['level'])
|
2019-03-19 19:13:56 +00:00
|
|
|
|
2020-01-07 17:23:18 +00:00
|
|
|
if task['worker-type'] in config.params['try_task_config'].get('worker-overrides', {}):
|
|
|
|
worker_pool = (
|
|
|
|
config.params['try_task_config']['worker-overrides'][task['worker-type']]
|
|
|
|
)
|
|
|
|
provisioner_id, worker_type = worker_pool.split('/', 1)
|
|
|
|
else:
|
|
|
|
provisioner_id, worker_type = get_worker_type(
|
|
|
|
config.graph_config,
|
|
|
|
task['worker-type'],
|
|
|
|
level=level,
|
|
|
|
release_level=config.params.release_level(),
|
|
|
|
)
|
2019-03-19 19:13:56 +00:00
|
|
|
task['worker-type'] = '/'.join([provisioner_id, worker_type])
|
2017-12-06 03:35:45 +00:00
|
|
|
project = config.params['project']
|
2016-09-12 15:51:49 +00:00
|
|
|
|
2016-08-31 15:24:54 +00:00
|
|
|
routes = task.get('routes', [])
|
2017-12-06 03:35:45 +00:00
|
|
|
scopes = [s.format(level=level, project=project) for s in task.get('scopes', [])]
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
|
|
|
# set up extra
|
2016-08-31 15:24:54 +00:00
|
|
|
extra = task.get('extra', {})
|
2017-09-22 19:48:33 +00:00
|
|
|
extra['parent'] = os.environ.get('TASK_ID', '')
|
2016-08-31 15:24:54 +00:00
|
|
|
task_th = task.get('treeherder')
|
|
|
|
if task_th:
|
2018-04-12 15:50:23 +00:00
|
|
|
extra.setdefault('treeherder-platform', task_th['platform'])
|
2016-08-31 15:24:54 +00:00
|
|
|
treeherder = extra.setdefault('treeherder', {})
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2016-08-31 15:24:54 +00:00
|
|
|
machine_platform, collection = task_th['platform'].split('/', 1)
|
|
|
|
treeherder['machine'] = {'platform': machine_platform}
|
|
|
|
treeherder['collection'] = {collection: True}
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2017-11-02 17:37:20 +00:00
|
|
|
group_names = config.graph_config['treeherder']['group-names']
|
2016-08-31 15:24:54 +00:00
|
|
|
groupSymbol, symbol = split_symbol(task_th['symbol'])
|
2016-09-12 15:51:49 +00:00
|
|
|
if groupSymbol != '?':
|
|
|
|
treeherder['groupSymbol'] = groupSymbol
|
2017-11-02 17:37:20 +00:00
|
|
|
if groupSymbol not in group_names:
|
2018-01-12 20:49:27 +00:00
|
|
|
path = os.path.join(config.path, task.get('job-from', ''))
|
|
|
|
raise Exception(UNKNOWN_GROUP_NAME.format(groupSymbol, path))
|
2017-11-02 17:37:20 +00:00
|
|
|
treeherder['groupName'] = group_names[groupSymbol]
|
2016-08-31 15:24:54 +00:00
|
|
|
treeherder['symbol'] = symbol
|
2017-10-12 20:03:32 +00:00
|
|
|
if len(symbol) > 25 or len(groupSymbol) > 25:
|
|
|
|
raise RuntimeError("Treeherder group and symbol names must not be longer than "
|
|
|
|
"25 characters: {} (see {})".format(
|
|
|
|
task_th['symbol'],
|
|
|
|
TC_TREEHERDER_SCHEMA_URL,
|
|
|
|
))
|
2016-08-31 15:24:54 +00:00
|
|
|
treeherder['jobKind'] = task_th['kind']
|
|
|
|
treeherder['tier'] = task_th['tier']
|
|
|
|
|
2018-03-07 22:23:00 +00:00
|
|
|
branch_rev = get_branch_rev(config)
|
2017-10-10 15:47:28 +00:00
|
|
|
|
2017-12-08 04:19:00 +00:00
|
|
|
routes.append(
|
|
|
|
'{}.v2.{}.{}.{}'.format(TREEHERDER_ROUTE_ROOT,
|
2016-08-31 15:24:54 +00:00
|
|
|
config.params['project'],
|
2018-03-07 22:23:00 +00:00
|
|
|
branch_rev,
|
2016-08-31 15:24:54 +00:00
|
|
|
config.params['pushlog_id'])
|
2017-12-08 04:19:00 +00:00
|
|
|
)
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2016-09-12 15:51:49 +00:00
|
|
|
if 'expires-after' not in task:
|
2018-04-13 16:52:06 +00:00
|
|
|
task['expires-after'] = '28 days' if config.params.is_try() else '1 year'
|
2016-09-12 15:51:49 +00:00
|
|
|
|
|
|
|
if 'deadline-after' not in task:
|
|
|
|
task['deadline-after'] = '1 day'
|
|
|
|
|
2017-05-05 20:22:03 +00:00
|
|
|
if 'priority' not in task:
|
2019-03-07 04:20:36 +00:00
|
|
|
task['priority'] = get_default_priority(config.graph_config, config.params['project'])
|
2017-05-05 20:22:03 +00:00
|
|
|
|
2017-01-25 21:04:37 +00:00
|
|
|
tags = task.get('tags', {})
|
2019-04-16 22:02:46 +00:00
|
|
|
attributes = task.get('attributes', {})
|
|
|
|
|
2017-08-16 19:52:18 +00:00
|
|
|
tags.update({
|
|
|
|
'createdForUser': config.params['owner'],
|
|
|
|
'kind': config.kind,
|
2017-09-22 19:48:33 +00:00
|
|
|
'label': task['label'],
|
2019-04-16 22:02:46 +00:00
|
|
|
'retrigger': 'true' if attributes.get('retrigger', False) else 'false'
|
2017-08-16 19:52:18 +00:00
|
|
|
})
|
2017-01-25 21:04:37 +00:00
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
task_def = {
|
|
|
|
'provisionerId': provisioner_id,
|
|
|
|
'workerType': worker_type,
|
|
|
|
'routes': routes,
|
|
|
|
'created': {'relative-datestamp': '0 seconds'},
|
|
|
|
'deadline': {'relative-datestamp': task['deadline-after']},
|
|
|
|
'expires': {'relative-datestamp': task['expires-after']},
|
|
|
|
'scopes': scopes,
|
|
|
|
'metadata': {
|
|
|
|
'description': task['description'],
|
|
|
|
'name': task['label'],
|
|
|
|
'owner': config.params['owner'],
|
2019-07-11 22:00:51 +00:00
|
|
|
'source': config.params.file_url(config.path, pretty=True),
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
},
|
|
|
|
'extra': extra,
|
2017-01-25 21:04:37 +00:00
|
|
|
'tags': tags,
|
2017-05-05 20:22:03 +00:00
|
|
|
'priority': task['priority'],
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
}
|
|
|
|
|
2018-01-08 15:14:26 +00:00
|
|
|
if task.get('requires', None):
|
|
|
|
task_def['requires'] = task['requires']
|
|
|
|
|
2017-01-13 20:46:43 +00:00
|
|
|
if task_th:
|
|
|
|
# link back to treeherder in description
|
|
|
|
th_push_link = 'https://treeherder.mozilla.org/#/jobs?repo={}&revision={}'.format(
|
2018-03-07 22:23:00 +00:00
|
|
|
config.params['project'], branch_rev)
|
2017-01-13 20:46:43 +00:00
|
|
|
task_def['metadata']['description'] += ' ([Treeherder push]({}))'.format(
|
|
|
|
th_push_link)
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
# add the payload and adjust anything else as required (e.g., scopes)
|
2019-07-02 17:05:21 +00:00
|
|
|
payload_builders[task['worker']['implementation']].builder(config, task, task_def)
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
|
2018-03-19 18:15:52 +00:00
|
|
|
# Resolve run-on-projects
|
|
|
|
build_platform = attributes.get('build_platform')
|
|
|
|
resolve_keyed_by(task, 'run-on-projects', item_name=task['label'],
|
|
|
|
**{'build-platform': build_platform})
|
2016-09-12 15:51:49 +00:00
|
|
|
attributes['run_on_projects'] = task.get('run-on-projects', ['all'])
|
2017-10-30 13:41:51 +00:00
|
|
|
attributes['always_target'] = task['always-target']
|
2017-12-06 03:35:45 +00:00
|
|
|
# This logic is here since downstream tasks don't always match their
|
|
|
|
# upstream dependency's shipping_phase.
|
2020-01-21 17:12:08 +00:00
|
|
|
# A text_type task['shipping-phase'] takes precedence, then
|
2017-12-06 03:35:45 +00:00
|
|
|
# an existing attributes['shipping_phase'], then fall back to None.
|
|
|
|
if task.get('shipping-phase') is not None:
|
|
|
|
attributes['shipping_phase'] = task['shipping-phase']
|
|
|
|
else:
|
|
|
|
attributes.setdefault('shipping_phase', None)
|
|
|
|
# shipping_product will always match the upstream task's
|
|
|
|
# shipping_product, so a pre-set existing attributes['shipping_product']
|
|
|
|
# takes precedence over task['shipping-product']. However, make sure
|
|
|
|
# we don't have conflicting values.
|
|
|
|
if task.get('shipping-product') and \
|
|
|
|
attributes.get('shipping_product') not in (None, task['shipping-product']):
|
|
|
|
raise Exception(
|
|
|
|
"{} shipping_product {} doesn't match task shipping-product {}!".format(
|
|
|
|
task['label'], attributes['shipping_product'], task['shipping-product']
|
|
|
|
)
|
|
|
|
)
|
|
|
|
attributes.setdefault('shipping_product', task['shipping-product'])
|
2016-09-12 15:51:49 +00:00
|
|
|
|
2017-04-14 22:30:42 +00:00
|
|
|
# Set MOZ_AUTOMATION on all jobs.
|
|
|
|
if task['worker']['implementation'] in (
|
|
|
|
'generic-worker',
|
|
|
|
'docker-worker',
|
|
|
|
):
|
|
|
|
payload = task_def.get('payload')
|
|
|
|
if payload:
|
|
|
|
env = payload.setdefault('env', {})
|
|
|
|
env['MOZ_AUTOMATION'] = '1'
|
|
|
|
|
2019-07-26 21:55:27 +00:00
|
|
|
# Set TASKCLUSTER_ROOT_URL on workers that don't set it
|
|
|
|
if provisioner_id == 'terraform-packet':
|
|
|
|
env['TASKCLUSTER_ROOT_URL'] = get_root_url(False)
|
|
|
|
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
yield {
|
|
|
|
'label': task['label'],
|
|
|
|
'task': task_def,
|
2016-09-12 15:51:49 +00:00
|
|
|
'dependencies': task.get('dependencies', {}),
|
2019-03-04 17:07:34 +00:00
|
|
|
'soft-dependencies': task.get('soft-dependencies', []),
|
2016-09-12 15:51:49 +00:00
|
|
|
'attributes': attributes,
|
2017-08-01 20:02:59 +00:00
|
|
|
'optimization': task.get('optimization', None),
|
2018-08-20 18:29:15 +00:00
|
|
|
'release-artifacts': task.get('release-artifacts', []),
|
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
|
|
|
}
|
2016-09-16 15:08:57 +00:00
|
|
|
|
|
|
|
|
2018-01-17 03:05:56 +00:00
|
|
|
@transforms.add
|
|
|
|
def chain_of_trust(config, tasks):
|
|
|
|
for task in tasks:
|
|
|
|
if task['task'].get('payload', {}).get('features', {}).get('chainOfTrust'):
|
|
|
|
image = task.get('dependencies', {}).get('docker-image')
|
|
|
|
if image:
|
|
|
|
cot = task['task'].setdefault('extra', {}).setdefault('chainOfTrust', {})
|
|
|
|
cot.setdefault('inputs', {})['docker-image'] = {
|
|
|
|
'task-reference': '<docker-image>'
|
|
|
|
}
|
|
|
|
yield task
|
|
|
|
|
|
|
|
|
2017-11-27 20:59:35 +00:00
|
|
|
@transforms.add
|
|
|
|
def check_task_identifiers(config, tasks):
|
|
|
|
"""Ensures that all tasks have well defined identifiers:
|
2019-02-22 06:09:26 +00:00
|
|
|
^[a-zA-Z0-9_-]{1,38}$
|
2017-11-27 20:59:35 +00:00
|
|
|
"""
|
2019-02-22 06:09:26 +00:00
|
|
|
e = re.compile("^[a-zA-Z0-9_-]{1,38}$")
|
2017-11-27 20:59:35 +00:00
|
|
|
for task in tasks:
|
2019-07-02 17:05:21 +00:00
|
|
|
for attrib in ('workerType', 'provisionerId'):
|
|
|
|
if not e.match(task['task'][attrib]):
|
2017-11-27 20:59:35 +00:00
|
|
|
raise Exception(
|
|
|
|
'task {}.{} is not a valid identifier: {}'.format(
|
2019-07-02 17:05:21 +00:00
|
|
|
task['label'], attrib, task['task'][attrib]))
|
2017-11-27 20:59:35 +00:00
|
|
|
yield task
|
|
|
|
|
|
|
|
|
2017-12-16 21:09:25 +00:00
|
|
|
@transforms.add
|
|
|
|
def check_task_dependencies(config, tasks):
|
|
|
|
"""Ensures that tasks don't have more than 100 dependencies."""
|
|
|
|
for task in tasks:
|
|
|
|
if len(task['dependencies']) > MAX_DEPENDENCIES:
|
|
|
|
raise Exception(
|
|
|
|
'task {}/{} has too many dependencies ({} > {})'.format(
|
|
|
|
config.kind, task['label'], len(task['dependencies']),
|
|
|
|
MAX_DEPENDENCIES))
|
|
|
|
yield task
|
|
|
|
|
|
|
|
|
2017-08-23 15:57:59 +00:00
|
|
|
def check_caches_are_volumes(task):
|
|
|
|
"""Ensures that all cache paths are defined as volumes.
|
|
|
|
|
|
|
|
Caches and volumes are the only filesystem locations whose content
|
|
|
|
isn't defined by the Docker image itself. Some caches are optional
|
|
|
|
depending on the job environment. We want paths that are potentially
|
|
|
|
caches to have as similar behavior regardless of whether a cache is
|
|
|
|
used. To help enforce this, we require that all paths used as caches
|
|
|
|
to be declared as Docker volumes. This check won't catch all offenders.
|
|
|
|
But it is better than nothing.
|
|
|
|
"""
|
2020-02-07 18:12:23 +00:00
|
|
|
volumes = set(six.ensure_text(s) for s in task['worker']['volumes'])
|
|
|
|
paths = set(six.ensure_text(c['mount-point'])
|
|
|
|
for c in task['worker'].get('caches', []))
|
2017-08-23 15:57:59 +00:00
|
|
|
missing = paths - volumes
|
|
|
|
|
|
|
|
if not missing:
|
|
|
|
return
|
|
|
|
|
|
|
|
raise Exception('task %s (image %s) has caches that are not declared as '
|
2018-12-05 19:19:56 +00:00
|
|
|
'Docker volumes: %s '
|
|
|
|
'(have you added them as VOLUMEs in the Dockerfile?)'
|
2018-01-31 20:20:11 +00:00
|
|
|
% (task['label'], task['worker']['docker-image'],
|
|
|
|
', '.join(sorted(missing))))
|
2017-08-23 15:57:59 +00:00
|
|
|
|
|
|
|
|
2020-03-05 23:22:13 +00:00
|
|
|
def check_required_volumes(task):
|
|
|
|
"""
|
|
|
|
Ensures that all paths that are required to be volumes are defined as volumes.
|
|
|
|
|
|
|
|
Performance of writing to files in poor in directories not marked as
|
|
|
|
volumes, in docker. Ensure that paths that are often written to are marked
|
|
|
|
as volumes.
|
|
|
|
"""
|
|
|
|
volumes = set(task['worker']['volumes'])
|
|
|
|
paths = set(task['worker'].get('required-volumes', []))
|
|
|
|
missing = paths - volumes
|
|
|
|
|
|
|
|
if not missing:
|
|
|
|
return
|
|
|
|
|
|
|
|
raise Exception('task %s (image %s) has paths that should be volumes for peformance '
|
|
|
|
'that are not declared as Docker volumes: %s '
|
|
|
|
'(have you added them as VOLUMEs in the Dockerfile?)'
|
|
|
|
% (task['label'], task['worker']['docker-image'],
|
|
|
|
', '.join(sorted(missing))))
|
|
|
|
|
|
|
|
|
2017-08-18 22:30:32 +00:00
|
|
|
@transforms.add
|
|
|
|
def check_run_task_caches(config, tasks):
|
|
|
|
"""Audit for caches requiring run-task.
|
|
|
|
|
|
|
|
run-task manages caches in certain ways. If a cache managed by run-task
|
|
|
|
is used by a non run-task task, it could cause problems. So we audit for
|
|
|
|
that and make sure certain cache names are exclusive to run-task.
|
|
|
|
|
|
|
|
IF YOU ARE TEMPTED TO MAKE EXCLUSIONS TO THIS POLICY, YOU ARE LIKELY
|
|
|
|
CONTRIBUTING TECHNICAL DEBT AND WILL HAVE TO SOLVE MANY OF THE PROBLEMS
|
|
|
|
THAT RUN-TASK ALREADY SOLVES. THINK LONG AND HARD BEFORE DOING THAT.
|
|
|
|
"""
|
|
|
|
re_reserved_caches = re.compile('''^
|
2019-02-21 07:32:36 +00:00
|
|
|
(checkouts|tooltool-cache)
|
2017-08-18 22:30:32 +00:00
|
|
|
''', re.VERBOSE)
|
|
|
|
|
2019-02-21 07:32:36 +00:00
|
|
|
re_sparse_checkout_cache = re.compile('^checkouts-sparse')
|
|
|
|
|
|
|
|
cache_prefix = '{trust_domain}-level-{level}-'.format(
|
|
|
|
trust_domain=config.graph_config['trust-domain'],
|
|
|
|
level=config.params['level'],
|
|
|
|
)
|
2017-08-24 01:54:14 +00:00
|
|
|
|
2017-08-18 22:30:32 +00:00
|
|
|
suffix = _run_task_suffix()
|
|
|
|
|
|
|
|
for task in tasks:
|
|
|
|
payload = task['task'].get('payload', {})
|
|
|
|
command = payload.get('command') or ['']
|
2017-08-24 01:54:14 +00:00
|
|
|
|
2020-01-21 17:12:08 +00:00
|
|
|
main_command = command[0] if isinstance(command[0], text_type) else ''
|
2017-08-24 01:54:14 +00:00
|
|
|
run_task = main_command.endswith('run-task')
|
|
|
|
|
|
|
|
require_sparse_cache = False
|
|
|
|
have_sparse_cache = False
|
|
|
|
|
|
|
|
if run_task:
|
|
|
|
for arg in command[1:]:
|
2020-01-21 17:12:08 +00:00
|
|
|
if not isinstance(arg, text_type):
|
2017-08-24 01:54:14 +00:00
|
|
|
continue
|
|
|
|
|
|
|
|
if arg == '--':
|
|
|
|
break
|
|
|
|
|
2018-12-31 18:52:26 +00:00
|
|
|
if arg.startswith('--gecko-sparse-profile'):
|
2018-11-26 19:27:22 +00:00
|
|
|
if '=' not in arg:
|
|
|
|
raise Exception(
|
2018-12-31 18:52:26 +00:00
|
|
|
'{} is specifying `--gecko-sparse-profile` to run-task '
|
|
|
|
'as two arguments. Unable to determine if the sparse '
|
|
|
|
'profile exists.'.format(
|
2018-11-26 19:27:22 +00:00
|
|
|
task['label']))
|
|
|
|
_, sparse_profile = arg.split('=', 1)
|
|
|
|
if not os.path.exists(os.path.join(GECKO, sparse_profile)):
|
|
|
|
raise Exception(
|
|
|
|
'{} is using non-existant sparse profile {}.'.format(
|
|
|
|
task['label'], sparse_profile))
|
2017-08-24 01:54:14 +00:00
|
|
|
require_sparse_cache = True
|
|
|
|
break
|
2017-08-18 22:30:32 +00:00
|
|
|
|
|
|
|
for cache in payload.get('cache', {}):
|
2019-02-21 07:32:36 +00:00
|
|
|
if not cache.startswith(cache_prefix):
|
|
|
|
raise Exception(
|
|
|
|
'{} is using a cache ({}) which is not appropriate '
|
|
|
|
'for its trust-domain and level. It should start with {}.'
|
|
|
|
.format(task['label'], cache, cache_prefix)
|
|
|
|
)
|
|
|
|
|
|
|
|
cache = cache[len(cache_prefix):]
|
|
|
|
|
2017-08-24 01:54:14 +00:00
|
|
|
if re_sparse_checkout_cache.match(cache):
|
|
|
|
have_sparse_cache = True
|
|
|
|
|
2017-08-18 22:30:32 +00:00
|
|
|
if not re_reserved_caches.match(cache):
|
|
|
|
continue
|
|
|
|
|
|
|
|
if not run_task:
|
|
|
|
raise Exception(
|
|
|
|
'%s is using a cache (%s) reserved for run-task '
|
|
|
|
'change the task to use run-task or use a different '
|
|
|
|
'cache name' % (task['label'], cache))
|
|
|
|
|
|
|
|
if not cache.endswith(suffix):
|
|
|
|
raise Exception(
|
|
|
|
'%s is using a cache (%s) reserved for run-task '
|
|
|
|
'but the cache name is not dependent on the contents '
|
|
|
|
'of run-task; change the cache name to conform to the '
|
|
|
|
'naming requirements' % (task['label'], cache))
|
|
|
|
|
2017-08-24 01:54:14 +00:00
|
|
|
if require_sparse_cache and not have_sparse_cache:
|
|
|
|
raise Exception('%s is using a sparse checkout but not using '
|
|
|
|
'a sparse checkout cache; change the checkout '
|
|
|
|
'cache name so it is sparse aware' % task['label'])
|
|
|
|
|
2017-08-18 22:30:32 +00:00
|
|
|
yield task
|