Bug 1737673 - Add documentation for the 'variants' system, r=jmaher

This also splits the blurb for the 'test' kind into a new file as it was
getting too long for kinds.rst. Having documentation for every kind in the same
file tends to lead towards uninformative documentation.

Differential Revision: https://phabricator.services.mozilla.com/D129830

taskcluster/docs/kinds.rst

@@ -150,40 +150,13 @@ spidermonkey portion.  Each task runs specific tests after the build.
 test
 ----
 
-The ``desktop-test`` kind defines tests for builds.  Its ``tests.yml`` defines
+See the :doc:`test kind documentation <kinds/test>` for more info.
-the full suite of desktop tests and their particulars, leaving it to the
-transforms to determine how those particulars apply to the various platforms.
 
-The process of generating tests goes like this, based on a set of YAML files
+.. toctree::
-named in ``kind.yml``:
+   :hidden:
 
- * For each build task, determine the related test platforms based on the build
+   kinds/test
-   platform.  For example, a Windows 2010 build might be tested on Windows 7
-   and Windows 10.  Each test platform specifies "test sets" indicating which
-   tests to run.  This is configured in the file named
-   ``test-platforms.yml``.
 
- * Each test set is expanded to a list of tests to run.  This is configured in
-   the file named by ``test-sets.yml``. A platform may specify several test
-   sets, in which case the union of those sets is used.
-
- * Each named test is looked up in the file named by ``tests.yml`` to find a
-   test description.  This test description indicates what the test does, how
-   it is reported to treeherder, and how to perform the test, all in a
-   platform-independent fashion.
-
- * Each test description is converted into one or more tasks.  This is
-   performed by a sequence of transforms defined in the ``transforms`` key in
-   ``kind.yml``.  See :doc:`transforms`: for more information on these
-   transforms.
-
- * The resulting tasks become a part of the task graph.
-
-.. important::
-
-    This process generates *all* test jobs, regardless of tree or try syntax.
-    It is up to a later stages of the task-graph generation (the target set and
-    optimization) to select the tests that will actually be performed.
 
 docker-image
 ------------

taskcluster/docs/kinds/test.rst

@@ -0,0 +1,79 @@
+Test Kind
+=========
+
+The ``test`` kind defines both desktop and mobile tests for builds. Each YAML
+file referenced in ``kind.yml`` defines the full set of tests for the
+associated suite.
+
+The process of generating tests goes like this, based on a set of YAML files
+named in ``kind.yml``:
+
+ * For each build task, determine the related test platforms based on the build
+   platform. For example, a Windows 2010 build might be tested on Windows 7
+   and Windows 10. Each test platform specifies "test sets" indicating which
+   tests to run. This is configured in the file named
+   ``test-platforms.yml``.
+
+ * Each test set is expanded to a list of tests to run.  This is configured in
+   the file named by ``test-sets.yml``. A platform may specify several test
+   sets, in which case the union of those sets is used.
+
+ * Each named test is looked up in the file named by ``tests.yml`` to find a
+   test description.  This test description indicates what the test does, how
+   it is reported to treeherder, and how to perform the test, all in a
+   platform-independent fashion.
+
+ * Each test description is converted into one or more tasks.  This is
+   performed by a sequence of transforms defined in the ``transforms`` key in
+   ``kind.yml``.  See :doc:`transforms`: for more information on these
+   transforms.
+
+ * The resulting tasks become a part of the task graph.
+
+.. important::
+
+    This process generates *all* test jobs, regardless of tree or try syntax.
+    It is up to a later stages of the task-graph generation (the target set and
+    optimization) to select the tests that will actually be performed.
+
+
+Variants
+--------
+
+Sometimes we want to run the same tests under a different Firefox context,
+usually this means with a pref set. The concept of ``variants`` was invented to
+handle this use case. A variant is a stanza of configuration that can be merged
+into each test definition. Variants are defined in the `variants.yml`_ file.
+See this file for an up to date list of active variants and the pref(s) they
+set.
+
+Each variant must conform to the
+:py:data:`~gecko_taskgraph.transforms.tests.variant_description_schema`:
+
+* **description** (required) - A description explaining what the variant is for
+* **suffix** (required) - A suffix to apply to the task label and treeherder symbol
+* **when** - A `json-e`_ expression that must evaluate to ``true`` for the variant
+  to be applied. The ``task`` definition is passed in as context
+* **replace** - A dictionary that will overwrite keys in the task definition
+* **merge** - A dictionary that will be merged into the task definition using
+  the :py:func:`~gecko_taskgraph.util.templates.merge` function.
+
+
+Defining Variants
+~~~~~~~~~~~~~~~~~
+
+Variants can be defined in the test YAML files using the ``variants`` key. E.g:
+
+.. code-block:: yaml
+
+   example-suite:
+       variants:
+           - foo
+           - bar
+
+This will split the task into three. The original task, the task with the
+config from the variant named 'foo' merged in and the task with the config from
+the variant named 'bar' merged in.
+
+.. _variants.yml: https://searchfox.org/mozilla-central/source/taskcluster/ci/test/variants.yml
+.. _json-e: https://json-e.js.org/

taskcluster/gecko_taskgraph/transforms/tests.py

@@ -572,10 +572,17 @@ variant_description_schema = Schema(
         }
     }
 )
+"""variant description schema"""
 
 
 @transforms.add
 def split_variants(config, tasks):
+    """Splits test definitions into multiple tasks based on the `variants` key.
+
+    If `variants` are defined, the original task will be yielded along with a
+    copy of the original task for each variant defined in the list. The copies
+    will have the 'unittest_variant' attribute set.
+    """
     validate_schema(variant_description_schema, TEST_VARIANTS, "In variants.yml:")
 
     for task in tasks: