This adds an optional "platforms" key to the job description. It can be used in conjunction with
"by-platform" like so:
platforms:
- linux
- windows
worker-type:
by-platform:
linux: ...
windows: ...
worker:
by-platform:
linux: ...
windows: ...
MozReview-Commit-ID: JwL1NAR4bnY
--HG--
extra : rebase_source : 637dd777912e256d82092ba3e0edd6937cbb03c6
Previously, we ran a single "target task" function to mutate the full
task graph into a subset based on input parameters (try syntax,
repository being built for, etc). This concept is useful. But
the implementation was limiting because we could only have a single
"target tasks" function.
This commit introduces the concept of "filters." They conceptually
do the same thing as "target tasks methods" but you can run more than
1 of them.
Filters are simply functions that examine an input graph+parameters
and emit nodes that should be retained. Filters, like target tasks
methods, are defined via decorated functions in a module.
TaskGraphGenerator has been converted to use filters. The list of
defined filters can be defined in the parameters dict passed into
TaskGraphGenerator. A default filter list is provided in decision.py.
The intent is to eventually convert target tasks to filters. Until
that happens, we always run the registered target tasks method via
a filter proxy function.
No new tests have been added because we don't yet have any
functionality relying explicitly on filters. Tests will be added in
a subsequent commit once we add a new filter.
While I was here, I also snuck in some logging on the size of the
graphs.
MozReview-Commit-ID: ERn2hIYbMRp
--HG--
extra : rebase_source : 36b8e86aa64b2f52b03b31b5497759b0009fb921
The `from_parameters` method was never used, and let do confusion over the role
of these parameters. Now there are only two, and they are always required.
MozReview-Commit-ID: AbPqijXucu5
--HG--
extra : rebase_source : 85affd063a543c549afaaa36ce7ee31ed1f943d5
ff5a4bab0813 (bug 1311791) and 332a08725ed0 (bug 1292071) changed
behavior of the VCS caches. First, the store cache / path was merged
into the checkouts cache. Then the path of the Mercurial shared store
was moved within the cache to always be rooted at the cache root.
Caches are shared across tasks. Tasks can execute on any revision
configured to use a cache. So, when interacting with caches, it is
important to consider how every revision configured to use that
cache will interact with it.
Take this scenario for example.
A worker executes a task where the hg shared store is rooted at
/home/worker/checkouts/src/hg-shared. Then the worker executes a
task where the hg shared store is rooted at
/home/worker/checkouts/hg-store. `hg robustcheckout` will see the
checkout from the first task. But then it sees that the store
it is pointing to is at an unexpected location
(checkouts/src/hg-shared instead of checkouts/hg-store). `hg
robustcheckout` aggressively normalizes state to ensure
consistency. So when it sees this mismatch, it blows away the
checkout and creates one from checkouts/hg-store to replace it.
That's a lot of overhead. And this cycle can repeat itself if
the right combination of revisions run on the worker!
A solution to this problem is to create a clean break from caches
when cache semantics change. In TaskCluster, that means using a
different cache.
This commit introduces a "version" component to the checkouts
cache name. By doing so, we create a clean break from all previous
caches, ensuring all revisions this point forward won't encounter
an hg shared store at an unexpected location. This also paves the
road for easily making additional clean breaks in the future.
MozReview-Commit-ID: JT8yuULKpch
--HG--
extra : rebase_source : 2e5d321e4314ff7543423d3c7837b770b7c8a3a3
This definitely isn't all of it. But I got the obvious chunks. Still
need to clean up mozharness.
MozReview-Commit-ID: JTZBydP3i2r
--HG--
extra : rebase_source : 782401359036751896ef6c153a81a06cb14031bb
We were seeing issues with the Mercurial working directory not being
pristine. While I can't reproduce this, I have a hunch it is due to
mixing and matching stores and checkouts in TaskCluster. For example,
if a worker supports running concurrent tasks and 2 tasks arrive at
the same time, the caches for the store and checkout may look like:
(store0, checkout0)
(store1, checkout1)
However, the next task may get:
(store1, checkout0)
This may confuse Mercurial.
This commit eliminates the "hg-shared" cache and places the shared
stores as a sibling directory of the checkout.
MozReview-Commit-ID: 8SzyS6wWf9C
--HG--
extra : rebase_source : 6205f3fe944a6ad2cb833a5a7c0ae5ba136eaea6
This patch prevents tests which have the 'run_on_projects' attribute assigned to an empty set from running when the try message contains '-p all' and '-u all' together. It also makes the 'match_test' function a little more readable and updates the 'attributes.rst' document to reflect the changes that were made.
MozReview-Commit-ID: IMk0cmSza8U
--HG--
extra : rebase_source : 06d542cbec16bd8091bc282ad1b831fb4671e72a
In particular, this makes a stronger push for readers to consult the source
files, which honestly have all the good stuff in them anyway.
MozReview-Commit-ID: 9dGWQw59h1L
--HG--
extra : rebase_source : fac2fe9324a6a35825ceece948d1960487a75aa5
This has some notes for future work on the task, but will work for now.
MozReview-Commit-ID: 7J4tQeKj3KJ
--HG--
extra : rebase_source : d171b5393fa340a47927e8b89ece93392ff23e56
The task description now includes
* flexible specification of index routes (this will get simpler once buildbot
and gecko.v1 routes are removed)
* "run-on-projects", indicating the projects on which this task should run
* "{level}" is allowed in workerTypes
* For the docker-worker/docker-engine worker implementations, "docker-image"
can have the form {in-tree: in-tree-name} to use an in-tree image. This was
previously implemented in the test transforms, but it is useful for other
tasks too!
* Optimizations, currently limited to "only-if-files-changed", can be specified
for each task.
* TreeHerder groupSymbol is optional
* expires-after and and deadline-after have default values (with the former
differing for try and non-try)
* coalesce-name triggers creation of both a coalesce route and a superseder URL
MozReview-Commit-ID: 70vtYs5lz5P
--HG--
extra : rebase_source : 9c557d68239f42466d9724d48ed5bf77648f9aa0
Rename to taskgraph.transforms.task.
This also adds some Required and Optional declarations to the schema to be explicit,
and adjusts the transform to handle treeherder being optional.
MozReview-Commit-ID: FuKYayvlwB9
--HG--
rename : taskcluster/taskgraph/transforms/make_task.py => taskcluster/taskgraph/transforms/task.py
extra : rebase_source : 0913aa8cdf153cd086a7786de957535e9b3a4ee8
MozReview-Commit-ID: Lrxi8t53nwy
If a developer adds '--rebuild N' to their try syntax they will get test jobs scheduled N times.
This is useful to determine intermittency rate.
This fixes a regression due to the recent refactoring on how we schedule tasks.
--HG--
extra : rebase_source : 355ca631353015bf63461c194168d753efd6958e
Currently, TaskCluster tasks tend to use the "workspace" directory as
a cache that manages the source checkout *and* additional state.
Historically at Mozilla, we've lumped "source checkout" and "workspace"
(sometimes known as an "objdir") into the same directory. This is
not ideal. Ideally, there is an immutable, read-only source checkout
and all files produced from that source live in a separate directory.
In this commit, the "workspace" directory for the "lint" image has been
renamed to "checkouts" and all tasks using the image have been updated
accordingly. By having "checkout" in the name, we clearly identify this
cache as being relevant to source checkouts, which IMO can serve a
different role from "workspaces." This distinction is important, as the
next commit will prevent the "checkouts" cache from getting optimized
out in certain tasks.
To hammer this point home, documentation on common caches has been
introduced.
MozReview-Commit-ID: BSEc4dM5YCt
--HG--
extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
A limitation of traditional docker build context generation is it
only includes files from the same directory as the Dockerfile. When
repositories have multiple, related Dockerfiles, this limitation
results file duplication or putting all Dockerfiles in the same
directory (which isn't feasible for mozilla-central since they would
need to be in the root directory).
This commit enhances Dockerfiles to allow *any* file from the
repository checkout to be ADDed to the docker build context.
Using the syntax "# %include <path>" you are able to include paths
or directories (relative from the top source directory root) in the
generated context archive. Files add this way are available under the
"topsrcdir/" path and can be ADDed to Docker images.
Since context archive generation is deterministic and the hash of
the resulting archive is used to determine when images need to be
rebuilt, any extra included file that changes will change the hash
of the context archive and force image regeneration.
Basic tests for the new feature have been added.
MozReview-Commit-ID: 4hPZesJuGQV
--HG--
extra : rebase_source : 99fae9fe82102126fbee879c134981047bb4a601
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
