mirror of
https://github.com/mozilla/gecko-dev.git
synced 2024-10-09 19:35:51 +00:00
Bug 1391776: cleanup of taskgraph docs; r=ahal
* eliminate heading for test kinds, of which there is now only one * make the caches document have a single heading in the TOC * break out mach commands into a separate document, add ./mach taskgraph morphed * remove docs for YAML templates support (the .yml file wasn't actually used -- I expect it was a merge leftover); these are still used for actions.yml, but once that is gone the code should be removed, too. * break try out into its own document, edit to distinguish "how to run try" from "how to generate config" MozReview-Commit-ID: 76ZopWA9TPL --HG-- extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
This commit is contained in:
Dustin J. Mitchell
parent
d69ec3ed28
commit
98a3631d7b
@ -1,23 +0,0 @@
|
||||
$inherits:
|
||||
from: 'build.yml'
|
||||
|
||||
|
||||
task:
|
||||
workerType: b2gbuild
|
||||
|
||||
scopes:
|
||||
# docker build tasks use tc-vcs so include the scope.
|
||||
- 'docker-worker:cache:level-{{level}}-{{project}}-tc-vcs'
|
||||
|
||||
payload:
|
||||
|
||||
cache:
|
||||
level-{{level}}-{{project}}-tc-vcs: '/home/worker/.tc-vcs'
|
||||
|
||||
# All docker builds share a common artifact directory for ease of uploading.
|
||||
artifacts:
|
||||
'public/build':
|
||||
type: directory
|
||||
path: '/home/worker/artifacts/'
|
||||
expires:
|
||||
relative-datestamp: '1 year'
|
||||
@ -16,7 +16,7 @@ multiple tasks requesting the same named cache, the worker will
|
||||
have multiple caches of the same name on the worker.
|
||||
|
||||
Caches and ``run-task``
|
||||
=======================
|
||||
-----------------------
|
||||
|
||||
``run-task`` is our generic task wrapper script. It does common activities
|
||||
like ensure a version control checkout is present.
|
||||
@ -39,10 +39,10 @@ Some caches are reserved for use with run-task. That property will be denoted
|
||||
below.
|
||||
|
||||
Common Caches
|
||||
=============
|
||||
-------------
|
||||
|
||||
Version Control Caches
|
||||
----------------------
|
||||
::::::::::::::::::::::
|
||||
|
||||
``level-{{level}}-checkouts-{{hash}}``
|
||||
This cache holds version control checkouts, each in a subdirectory named
|
||||
@ -77,7 +77,7 @@ Version Control Caches
|
||||
should be replaced with ``hg robustcheckout``.
|
||||
|
||||
Workspace Caches
|
||||
----------------
|
||||
::::::::::::::::
|
||||
|
||||
``level-{{level}}-*-workspace``
|
||||
These caches (of various names typically ending with ``workspace``)
|
||||
@ -85,7 +85,7 @@ Workspace Caches
|
||||
dependent on the task.
|
||||
|
||||
Other
|
||||
-----
|
||||
:::::
|
||||
|
||||
``level-{{level}}-tooltool-cache-{{hash}}
|
||||
Tooltool invocations should use this cache. Tooltool will store files here
|
||||
|
||||
@ -31,30 +31,22 @@ Hacking Task Graphs
|
||||
|
||||
The recommended process for changing task graphs is this:
|
||||
|
||||
1. Find a recent decision task on the project or branch you are working on, and
|
||||
download its ``parameters.yml`` artifact. Alternately, you
|
||||
can simply take note of the artifact URL, or just the decision task's
|
||||
``task-id``. This file contains all of the inputs to the task-graph
|
||||
generation process. Its contents are simple enough if you would like to
|
||||
modify it, and it is documented in :doc:`parameters`.
|
||||
|
||||
2. Run one of the ``mach taskgraph`` subcommands (see :doc:`taskgraph`) to
|
||||
generate a baseline against which to measure your changes, passing the
|
||||
parameters you found in the previous step. For example:
|
||||
1. Run one of the ``mach taskgraph`` subcommands (see :doc:`mach`) to
|
||||
generate a baseline against which to measure your changes.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
./mach taskgraph tasks --json -p parameters.yml > old-tasks.json
|
||||
./mach taskgraph tasks --json -p url/to/parameters.yml > old-tasks.json
|
||||
./mach taskgraph tasks --json -p task-id=<task-id> > old-tasks.json
|
||||
./mach taskgraph tasks --json > old-tasks.json
|
||||
|
||||
3. Make your modifications under ``taskcluster/``.
|
||||
2. Make your modifications under ``taskcluster/``.
|
||||
|
||||
4. Run the same ``mach taskgraph`` command, sending the output to a new file,
|
||||
3. Run the same ``mach taskgraph`` command, sending the output to a new file,
|
||||
and use ``diff`` to compare the old and new files. Make sure your changes
|
||||
have the desired effect and no undesirable side-effects.
|
||||
have the desired effect and no undesirable side-effects. A plain unified
|
||||
diff should be useful for most changes, but in some cases it may be helpful
|
||||
to post-process the JSON to strip distracting changes.
|
||||
|
||||
5. When you are satisfied with the changes, push them to try to ensure that the
|
||||
4. When you are satisfied with the changes, push them to try to ensure that the
|
||||
modified tasks work as expected.
|
||||
|
||||
Hacking Actions
|
||||
@ -67,8 +59,7 @@ If you are working on an action task and wish to test it out locally, use the
|
||||
|
||||
./mach taskgraph test-action-task \
|
||||
--task-id I4gu9KDmSZWu3KHx6ba6tw --task-group-id sMO4ybV9Qb2tmcI1sDHClQ \
|
||||
-p parameters.yml --input input.yml \
|
||||
hello_world_action
|
||||
--input input.yml hello_world_action
|
||||
|
||||
This invocation will run the hello world callback with the given inputs and
|
||||
print any created tasks to stdout, rather than actually creating them.
|
||||
@ -254,110 +245,3 @@ If you make another change not described here that turns out to be simple or
|
||||
common, please include an update to this file in your patch.
|
||||
|
||||
|
||||
Schedule a Task on Try
|
||||
----------------------
|
||||
|
||||
There are two methods for scheduling a task on try.
|
||||
|
||||
The first method is a command line string called ``try syntax`` which is passed
|
||||
into the decision task via the commit message. An example try syntax might look
|
||||
like:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
try: -b o -p linux64 -u mochitest-1 -t none
|
||||
|
||||
This gets parsed by ``taskgraph.try_option_syntax:TryOptionSyntax`` and returns
|
||||
a list of matching task labels. For more information see the
|
||||
`TryServer wiki page <https://wiki.mozilla.org/Try>`_.
|
||||
|
||||
The second method uses a checked-in file called ``try_task_config.json`` which
|
||||
lives at the root of the source dir. The format of this file is either a list
|
||||
of task labels, or a JSON object where task labels make up the keys. For
|
||||
example, the ``try_task_config.json`` file might look like:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
{
|
||||
"tasks": [
|
||||
"test-windows10-64/opt-web-platform-tests-12",
|
||||
"test-windows7-32/opt-reftest-1",
|
||||
"test-windows7-32/opt-reftest-2",
|
||||
"test-windows7-32/opt-reftest-3",
|
||||
"build-linux64/debug",
|
||||
"source-test-mozlint-eslint"
|
||||
]
|
||||
}
|
||||
|
||||
Very simply, this will run any task label that gets passed in as well as their
|
||||
dependencies. While it is possible to manually commit this file and push to
|
||||
try, it is mainly meant to be a generation target for various `tryselect`_
|
||||
choosers.
|
||||
|
||||
A list of all possible task labels can be obtained by running:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
$ ./mach taskgraph tasks
|
||||
|
||||
A list of task labels relevant to a tree (defaults to mozilla-central) can be
|
||||
obtained with:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
$ ./mach taskgraph target
|
||||
|
||||
Modifying Task Behavior on Try
|
||||
``````````````````````````````
|
||||
|
||||
It's possible to alter the definition of a task with templates. Templates are
|
||||
`JSON-e`_ files that live in the `taskgraph module`_. Templates can be specified
|
||||
from the ``try_task_config.json`` like this:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
{
|
||||
"tasks": [...],
|
||||
"templates": {
|
||||
artifact: {"enabled": 1}
|
||||
}
|
||||
}
|
||||
|
||||
Each key in the templates object denotes a new template to apply, and the value
|
||||
denotes extra context to use while rendering. When specified, a template will
|
||||
be applied to every task no matter what. If the template should only be applied
|
||||
to certain kinds of tasks, this needs to be specified in the template itself
|
||||
using JSON-e `condition statements`_.
|
||||
|
||||
The context available to the JSON-e render aims to match that of ``actions``.
|
||||
It looks like this:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
{
|
||||
"task": {
|
||||
"payload": {
|
||||
"env": { ... },
|
||||
...
|
||||
}
|
||||
"extra": {
|
||||
"treeherder": { ... },
|
||||
...
|
||||
},
|
||||
"tags": { "kind": "<kind>", ... },
|
||||
...
|
||||
},
|
||||
"input": {
|
||||
"enabled": 1,
|
||||
...
|
||||
},
|
||||
"taskId": "<task id>"
|
||||
}
|
||||
|
||||
See the `existing templates`_ for examples.
|
||||
|
||||
.. _tryselect: https://dxr.mozilla.org/mozilla-central/source/tools/tryselect
|
||||
.. _JSON-e: https://taskcluster.github.io/json-e/
|
||||
.. _taskgraph module: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
|
||||
.. _condition statements: https://taskcluster.github.io/json-e/#%60$if%60%20-%20%60then%60%20-%20%60else%60
|
||||
.. _existing templates: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
|
||||
|
||||
@ -22,12 +22,13 @@ check out the :doc:`how-to section <how-tos>`.
|
||||
.. toctree::
|
||||
|
||||
taskgraph
|
||||
mach
|
||||
loading
|
||||
transforms
|
||||
optimization
|
||||
yaml-templates
|
||||
docker-images
|
||||
cron
|
||||
try
|
||||
actions
|
||||
how-tos
|
||||
reference
|
||||
|
||||
@ -98,12 +98,15 @@ spidermonkey
|
||||
Spidermonkey tasks check out the full gecko source tree, then compile only the
|
||||
spidermonkey portion. Each task runs specific tests after the build.
|
||||
|
||||
Tests
|
||||
-----
|
||||
test
|
||||
----
|
||||
|
||||
Test tasks for Gecko products are divided into several kinds, but share a
|
||||
common implementation. The process goes like this, based on a set of YAML
|
||||
files named in ``kind.yml``:
|
||||
The ``desktop-test`` kind defines tests for builds. Its ``tests.yml`` defines
|
||||
the full suite of desktop tests and their particulars, leaving it to the
|
||||
transforms to determine how those particulars apply to the various platforms.
|
||||
|
||||
The process of generating tests goes like this, based on a set of YAML files
|
||||
named in ``kind.yml``:
|
||||
|
||||
* For each build task, determine the related test platforms based on the build
|
||||
platform. For example, a Windows 2010 build might be tested on Windows 7
|
||||
@ -130,17 +133,8 @@ files named in ``kind.yml``:
|
||||
.. important::
|
||||
|
||||
This process generates *all* test jobs, regardless of tree or try syntax.
|
||||
It is up to a later stage of the task-graph generation (the target set) to
|
||||
select the tests that will actually be performed.
|
||||
|
||||
test
|
||||
....
|
||||
|
||||
The ``desktop-test`` kind defines tests for builds. Its ``tests.yml`` defines
|
||||
the full suite of desktop tests and their particulars, leaving it to the
|
||||
transforms to determine how those particulars apply to the various platforms.
|
||||
|
||||
This kind includes both unit tests and talos.
|
||||
It is up to a later stages of the task-graph generation (the target set and
|
||||
optimization) to select the tests that will actually be performed.
|
||||
|
||||
docker-image
|
||||
------------
|
||||
|
||||
110
taskcluster/docs/mach.rst
Normal file
110
taskcluster/docs/mach.rst
Normal file
@ -0,0 +1,110 @@
|
||||
Mach commands
|
||||
=============
|
||||
|
||||
A number of mach subcommands are available aside from ``mach taskgraph
|
||||
decision`` to make this complex system more accessible to those trying to
|
||||
understand or modify it. They allow you to run portions of the
|
||||
graph-generation process and output the results.
|
||||
|
||||
``mach taskgraph tasks``
|
||||
Get the full task set
|
||||
|
||||
``mach taskgraph full``
|
||||
Get the full task graph
|
||||
|
||||
``mach taskgraph target``
|
||||
Get the target task set
|
||||
|
||||
``mach taskgraph target-graph``
|
||||
Get the target task graph
|
||||
|
||||
``mach taskgraph optimized``
|
||||
Get the optimized task graph
|
||||
|
||||
``mach taskgraph morphed``
|
||||
Get the morhped task graph
|
||||
|
||||
Each of these commands takes an optional ``--parameters`` option giving a file
|
||||
with parameters to guide the graph generation. The decision task helpfully
|
||||
produces such a file on every run, and that is generally the easiest way to get
|
||||
a parameter file. The parameter keys and values are described in
|
||||
:doc:`parameters`; using that information, you may modify an existing
|
||||
``parameters.yml`` or create your own. The ``--parameters`` option can also
|
||||
take an argument of the form ``project=<project>`` which will fetch the
|
||||
parameters from the latest push on that project; or ``task-id=<task-id>`` which
|
||||
will fetch the parameters from the given decision task. It defaults to
|
||||
``project=mozilla-central``.
|
||||
|
||||
See :doc:`how-tos` for further practical tips on debugging task-graph mechanics
|
||||
locally.
|
||||
|
||||
Taskgraph JSON Format
|
||||
---------------------
|
||||
By default, the above commands will only output a list of tasks. Use `-J` flag
|
||||
to output full task definitions. For example:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ ./mach taskgraph optimized -J
|
||||
|
||||
|
||||
Task graphs -- both the graph artifacts produced by the decision task and those
|
||||
output by the ``--json`` option to the ``mach taskgraph`` commands -- are JSON
|
||||
objects, keyed by label, or for optimized task graphs, by taskId. For
|
||||
convenience, the decision task also writes out ``label-to-taskid.json``
|
||||
containing a mapping from label to taskId. Each task in the graph is
|
||||
represented as a JSON object.
|
||||
|
||||
Each task has the following properties:
|
||||
|
||||
``kind``
|
||||
The name of this task's kind
|
||||
|
||||
``task_id``
|
||||
The task's taskId (only for optimized task graphs)
|
||||
|
||||
``label``
|
||||
The task's label
|
||||
|
||||
``attributes``
|
||||
The task's attributes
|
||||
|
||||
``dependencies``
|
||||
The task's in-graph dependencies, represented as an object mapping
|
||||
dependency name to label (or to taskId for optimized task graphs)
|
||||
|
||||
``optimizations``
|
||||
The optimizations to be applied to this task
|
||||
|
||||
``task``
|
||||
The task's TaskCluster task definition.
|
||||
|
||||
The results from each command are in the same format, but with some differences
|
||||
in the content:
|
||||
|
||||
* The ``tasks`` and ``target`` subcommands both return graphs with no edges.
|
||||
That is, just collections of tasks without any dependencies indicated.
|
||||
|
||||
* The ``optimized`` subcommand returns tasks that have been assigned taskIds.
|
||||
The dependencies array, too, contains taskIds instead of labels, with
|
||||
dependencies on optimized tasks omitted. However, the ``task.dependencies``
|
||||
array is populated with the full list of dependency taskIds. All task
|
||||
references are resolved in the optimized graph.
|
||||
|
||||
The output of the ``mach taskgraph`` commands are suitable for processing with
|
||||
the `jq <https://stedolan.github.io/jq/>`_ utility. For example, to extract all
|
||||
tasks' labels and their dependencies:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
jq 'to_entries | map({label: .value.label, dependencies: .value.dependencies})'
|
||||
|
||||
An alternate way of searching the output of ``mach taskgraph`` is
|
||||
`gron <https://github.com/tomnomnom/gron>`_, which converts json into a format
|
||||
that's easily searched with ``grep``
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
gron taskgraph.json | grep -E 'test.*machine.platform = "linux64";'
|
||||
./mach taskgraph --json | gron | grep ...
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
======================
|
||||
TaskGraph Mach Command
|
||||
======================
|
||||
========
|
||||
Overview
|
||||
========
|
||||
|
||||
The task graph is built by linking different kinds of tasks together, pruning
|
||||
out tasks that are not required, then optimizing by replacing subgraphs with
|
||||
@ -150,45 +150,6 @@ Action Tasks are currently scheduled by
|
||||
[pulse_actions](https://github.com/mozilla/pulse_actions). This feature is only
|
||||
present on ``try`` pushes for now.
|
||||
|
||||
Mach commands
|
||||
-------------
|
||||
|
||||
A number of mach subcommands are available aside from ``mach taskgraph
|
||||
decision`` to make this complex system more accessible to those trying to
|
||||
understand or modify it. They allow you to run portions of the
|
||||
graph-generation process and output the results.
|
||||
|
||||
``mach taskgraph tasks``
|
||||
Get the full task set
|
||||
|
||||
``mach taskgraph full``
|
||||
Get the full task graph
|
||||
|
||||
``mach taskgraph target``
|
||||
Get the target task set
|
||||
|
||||
``mach taskgraph target-graph``
|
||||
Get the target task graph
|
||||
|
||||
``mach taskgraph optimized``
|
||||
Get the optimized task graph
|
||||
|
||||
Each of these commands taskes a ``--parameters`` option giving a file with
|
||||
parameters to guide the graph generation. The decision task helpfully produces
|
||||
such a file on every run, and that is generally the easiest way to get a
|
||||
parameter file. The parameter keys and values are described in
|
||||
:doc:`parameters`; using that information, you may modify an existing
|
||||
``parameters.yml`` or create your own.
|
||||
|
||||
By default, the above commands will only output a list of tasks. Use `-J` flag
|
||||
to output full task definitions. For example:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ ./mach taskgraph optimized -J -p ~/Downloads/parameters.yml
|
||||
|
||||
See :doc:`how-tos` for further practical tips.
|
||||
|
||||
Task Parameterization
|
||||
---------------------
|
||||
|
||||
@ -209,66 +170,3 @@ using simple parameterized values, as follows:
|
||||
Multiple labels may be substituted in a single string, and ``<<>`` can be
|
||||
used to escape a literal ``<``.
|
||||
|
||||
Taskgraph JSON Format
|
||||
---------------------
|
||||
|
||||
Task graphs -- both the graph artifacts produced by the decision task and those
|
||||
output by the ``--json`` option to the ``mach taskgraph`` commands -- are JSON
|
||||
objects, keyed by label, or for optimized task graphs, by taskId. For
|
||||
convenience, the decision task also writes out ``label-to-taskid.json``
|
||||
containing a mapping from label to taskId. Each task in the graph is
|
||||
represented as a JSON object.
|
||||
|
||||
Each task has the following properties:
|
||||
|
||||
``kind``
|
||||
The name of this task's kind
|
||||
|
||||
``task_id``
|
||||
The task's taskId (only for optimized task graphs)
|
||||
|
||||
``label``
|
||||
The task's label
|
||||
|
||||
``attributes``
|
||||
The task's attributes
|
||||
|
||||
``dependencies``
|
||||
The task's in-graph dependencies, represented as an object mapping
|
||||
dependency name to label (or to taskId for optimized task graphs)
|
||||
|
||||
``optimizations``
|
||||
The optimizations to be applied to this task
|
||||
|
||||
``task``
|
||||
The task's TaskCluster task definition.
|
||||
|
||||
The results from each command are in the same format, but with some differences
|
||||
in the content:
|
||||
|
||||
* The ``tasks`` and ``target`` subcommands both return graphs with no edges.
|
||||
That is, just collections of tasks without any dependencies indicated.
|
||||
|
||||
* The ``optimized`` subcommand returns tasks that have been assigned taskIds.
|
||||
The dependencies array, too, contains taskIds instead of labels, with
|
||||
dependencies on optimized tasks omitted. However, the ``task.dependencies``
|
||||
array is populated with the full list of dependency taskIds. All task
|
||||
references are resolved in the optimized graph.
|
||||
|
||||
The output of the ``mach taskgraph`` commands are suitable for processing with
|
||||
the `jq <https://stedolan.github.io/jq/>`_ utility. For example, to extract all
|
||||
tasks' labels and their dependencies:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
jq 'to_entries | map({label: .value.label, dependencies: .value.dependencies})'
|
||||
|
||||
An alternate way of searching the output of ``mach taskgraph`` is
|
||||
`gron <https://github.com/tomnomnom/gron>`_, which converts json into a format
|
||||
that's easily searched with ``grep``
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
gron taskgraph.json | grep -E 'test.*machine.platform = "linux64";'
|
||||
./mach taskgraph -p parameters.yml --json | gron | grep ...
|
||||
|
||||
|
||||
140
taskcluster/docs/try.rst
Normal file
140
taskcluster/docs/try.rst
Normal file
@ -0,0 +1,140 @@
|
||||
Try
|
||||
===
|
||||
|
||||
"Try" is a way to "try out" a proposed change safely before review, without
|
||||
officialy landing it. This functionality has been around for a *long* time in
|
||||
various forms, and can sometimes show its age.
|
||||
|
||||
Access to "push to try" is typically avilable to a much larger group of
|
||||
developers than those who can land changes in integration and release branches.
|
||||
Specifically, try pushes are allowed for anyone with `SCM Level`_ 1, while
|
||||
integration branches are at SCM level 3.
|
||||
|
||||
Scheduling a Task on Try
|
||||
------------------------
|
||||
|
||||
There are two methods for scheduling a task on try.
|
||||
|
||||
Try Option Syntax
|
||||
:::::::::::::::::
|
||||
|
||||
The first, older method is a command line string called ``try syntax`` which is passed
|
||||
into the decision task via the commit message. The resulting commit is then
|
||||
pushed to the https://hg.mozilla.org/try repository. An example try syntax
|
||||
might look like:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
try: -b o -p linux64 -u mochitest-1 -t none
|
||||
|
||||
This gets parsed by ``taskgraph.try_option_syntax:TryOptionSyntax`` and returns
|
||||
a list of matching task labels. For more information see the
|
||||
`TryServer wiki page <https://wiki.mozilla.org/Try>`_.
|
||||
|
||||
Try Task Config
|
||||
:::::::::::::::
|
||||
|
||||
The second, more modern method specifies exactly the tasks to run. That list
|
||||
of tasks is usually generated locally with some `local tool <tryselect>`_ and
|
||||
attached to the commit pushed to the try repository. This gives finer control
|
||||
over exactly what runs and enables growth of an ecosystem of tooling
|
||||
appropriate to varied circumstances.
|
||||
|
||||
Implementation
|
||||
,,,,,,,,,,,,,,
|
||||
|
||||
This method uses a checked-in file called ``try_task_config.json`` which lives
|
||||
at the root of the source dir. The JSON object in this file contains a
|
||||
``tasks`` key giving the labels of the tasks to run. For example, the
|
||||
``try_task_config.json`` file might look like:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
{
|
||||
"tasks": [
|
||||
"test-windows10-64/opt-web-platform-tests-12",
|
||||
"test-windows7-32/opt-reftest-1",
|
||||
"test-windows7-32/opt-reftest-2",
|
||||
"test-windows7-32/opt-reftest-3",
|
||||
"build-linux64/debug",
|
||||
"source-test-mozlint-eslint"
|
||||
]
|
||||
}
|
||||
|
||||
Very simply, this will run any task label that gets passed in as well as their
|
||||
dependencies. While it is possible to manually commit this file and push to
|
||||
try, it is mainly meant to be a generation target for various `tryselect`_
|
||||
choosers. For example:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
$ ./mach try fuzzy
|
||||
|
||||
A list of all possible task labels can be obtained by running:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
$ ./mach taskgraph tasks
|
||||
|
||||
A list of task labels relevant to a tree (defaults to mozilla-central) can be
|
||||
obtained with:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
$ ./mach taskgraph target
|
||||
|
||||
Modifying Tasks in a Try Push
|
||||
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
|
||||
|
||||
It's possible to alter the definition of a task with templates. Templates are
|
||||
`JSON-e`_ files that live in the `taskgraph module`_. Templates can be specified
|
||||
from the ``try_task_config.json`` like this:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
{
|
||||
"tasks": [...],
|
||||
"templates": {
|
||||
artifact: {"enabled": 1}
|
||||
}
|
||||
}
|
||||
|
||||
Each key in the templates object denotes a new template to apply, and the value
|
||||
denotes extra context to use while rendering. When specified, a template will
|
||||
be applied to every task no matter what. If the template should only be applied
|
||||
to certain kinds of tasks, this needs to be specified in the template itself
|
||||
using JSON-e `condition statements`_.
|
||||
|
||||
The context available to the JSON-e render aims to match that of ``actions``.
|
||||
It looks like this:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
{
|
||||
"task": {
|
||||
"payload": {
|
||||
"env": { ... },
|
||||
...
|
||||
}
|
||||
"extra": {
|
||||
"treeherder": { ... },
|
||||
...
|
||||
},
|
||||
"tags": { "kind": "<kind>", ... },
|
||||
...
|
||||
},
|
||||
"input": {
|
||||
"enabled": 1,
|
||||
...
|
||||
},
|
||||
"taskId": "<task id>"
|
||||
}
|
||||
|
||||
See the `existing templates`_ for examples.
|
||||
|
||||
.. _tryselect: https://dxr.mozilla.org/mozilla-central/source/tools/tryselect
|
||||
.. _JSON-e: https://taskcluster.github.io/json-e/
|
||||
.. _taskgraph module: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
|
||||
.. _condition statements: https://taskcluster.github.io/json-e/#%60$if%60%20-%20%60then%60%20-%20%60else%60
|
||||
.. _existing templates: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
|
||||
.. _SCM Level: https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/
|
||||
@ -1,49 +0,0 @@
|
||||
Task Definition YAML Templates
|
||||
==============================
|
||||
|
||||
A few kinds of tasks are described using templated YAML files. These files
|
||||
allow some limited forms of inheritance and template substitution as well as
|
||||
the usual YAML features, as described below.
|
||||
|
||||
Please do not use these features in new kinds. If you are tempted to use
|
||||
variable substitution over a YAML file to define tasks, please instead
|
||||
implement a new kind-specific transform to accopmlish your goal. For example,
|
||||
if the current push-id must be included as an argument in
|
||||
``task.payload.command``, write a transform function that makes that assignment
|
||||
while building a job description, rather than parameterizing that value in the
|
||||
input to the transforms.
|
||||
|
||||
Inheritance
|
||||
-----------
|
||||
|
||||
One YAML file can "inherit" from another by including a top-level ``$inherits``
|
||||
key. That key specifies the parent file in ``from``, and optionally a
|
||||
collection of variables in ``variables``. For example:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
$inherits:
|
||||
from: 'tasks/builds/base_linux32.yml'
|
||||
variables:
|
||||
build_name: 'linux32'
|
||||
build_type: 'dbg'
|
||||
|
||||
Inheritance proceeds as follows: First, the child document has its template
|
||||
substitutions performed and is parsed as YAML. Then, the parent document is
|
||||
parsed, with substitutions specified by ``variables`` added to the template
|
||||
substitutions. Finally, the child document is merged with the parent.
|
||||
|
||||
To merge two JSON objects (dictionaries), each value is merged individually.
|
||||
Lists are merged by concatenating the lists from the parent and child
|
||||
documents. Atomic values (strings, numbers, etc.) are merged by preferring the
|
||||
child document's value.
|
||||
|
||||
Substitution
|
||||
------------
|
||||
|
||||
Each document is expanded using the PyStache template engine before it is
|
||||
parsed as YAML. The parameters for this expansion are specific to the task
|
||||
kind.
|
||||
|
||||
Simple value substitution looks like ``{{variable}}``. Function calls look
|
||||
like ``{{#function}}argument{{/function}}``.
|
||||
@ -221,6 +221,7 @@ def write_artifact(filename, data):
|
||||
|
||||
|
||||
def get_action_yml(parameters):
|
||||
# NOTE: when deleting this function, delete taskcluster/taskgraph/util/templates.py too
|
||||
templates = Templates(os.path.join(GECKO, "taskcluster/taskgraph"))
|
||||
action_parameters = parameters.copy()
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user