`run-task` is taught a --sparse-profile argument to be passed down
to `hg robustcheckout` for the main source checkout. It does what
you expect: performs a sparse checkout using the named profile.
The Taskgraph YAML for run-task is taught a "sparse-profile"
property to define the sparse profile. When defined, --sparse-profile
will be passed down to `run-task` and the cache name will be updated
to reflect the use of sparse checkout.
Our cache checking transform is updated to audit for the use of
--sparse-profile without the corresponding "-sparse" cache name
variation.
The reason we need a distinct cache name for sparse is because
clients that aren't sparse aware will be unable to read checkouts
that are sparse. By forcing sparse and non-sparse into different
cache pools, we avoid compatibility issues.
In the ideal world, we probably support sparse profiles on all the
VCS checkouts that `run-task` supports (e.g. --tools-checkout).
Perfect is the enemy of done. All of this is defined in-tree and
it is easy enough to change atomically.
MozReview-Commit-ID: 79k7Vul0hHO
--HG--
extra : rebase_source : babe9b42e2796c2341bffc6ecfe829f4daff9e0f
* eliminate heading for test kinds, of which there is now only one
* make the caches document have a single heading in the TOC
* break out mach commands into a separate document, add ./mach taskgraph morphed
* remove docs for YAML templates support (the .yml file wasn't actually
used -- I expect it was a merge leftover); these are still used for actions.yml,
but once that is gone the code should be removed, too.
* break try out into its own document, edit to distinguish "how to run try"
from "how to generate config"
MozReview-Commit-ID: 76ZopWA9TPL
--HG--
extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
Caches shared across levels scare me, even if readers are purported to
perform content verification. We shouldn't take any risks with released
Firefox builds being contaminated by e.g. Try tasks.
Also, the old cache name interferes with my desire to make cache
names dynamic. This requires dynamic scopes. We already have
have level-{{level}}-* scopes for caches. So having all caches
prefixed with this makes things flexible.
MozReview-Commit-ID: LsrcxIYoEh1
--HG--
extra : rebase_source : dfe97f92a726059200ed79afe215ef2cf1fd7bf1
This change adds an upload-generated-sources task kind that runs after nightly
builds, fetches their `target.generated-files.tar.gz` artifact, and uploads
all the contained files to an S3 bucket. For actual nightly and release builds
on SCM level 3 trees, the S3 bucket is configured to be publicly accessible,
so that tools like Socorro will be able to fetch generated source files that
appear in crash reports, and debuggers will be able to fetch generated sources
when they show up while debugging Nightly or Release builds.
There are also level-2 and level-1 S3 buckets configured for builds happening
on trees of other levels such as try. They are not configured as publicly
accessible, but they exist so that these tasks can be tested in try.
MozReview-Commit-ID: Js1HRftbtep
--HG--
extra : rebase_source : b1172c9cc8b8be437d3b94a6bf0ff6b2f7d3508b
extra : source : 73bf88110b3821d62a3d393e85b56896a12f2930
This provides a mechanism to modify the behaviour of tasks from a try push. The try_task_config.json
looks something like:
{
"tasks": ["build-linux64/opt", "test-linux64/opt-mochitest-e10s-1"],
"templates": {
"artifact": {"enabled": 1}
}
}
This tells taskgraph to apply the 'artifact' template to all tasks. Templates are JSONe based
.yml files that live under taskcluster/taskgraph/templates. Taskgraph will render every template
against every task definition. The templates themselves can then use JSONe condition statements to
filter out which tasks they should or shouldn't apply to.
MozReview-Commit-ID: J8HVZzOt4mX
--HG--
extra : rebase_source : 95a78bc56d3f90ff1b34aabd84ed92aff1e3d954
It is desirable for e.g. smooth toolchain transitions, to be able to
refer them with generic name from toolchain jobs, while they have more
specific names, including version numbers. For example, in a near
future, there could be a linux64-gcc-4.9 toolchain and a linux64-gcc-6.
The default would be former, but at some point we'd want to switch to
the latter, without having to change all the toolchain definitions.
Moreover, when the switch happens, it would be desirable to have some
jobs stick with the old version, which is hard to keep track of when
all the toolchain definitions for build jobs use the same versioned
toolchain. With an alias, jobs that want the default use the alias, and
jobs that want to use a specific version use the versioned toolchain
name.
--HG--
extra : rebase_source : 467d713edd00dbe358483f5ee749fa56900714dd
This introduces a 'try_task_config' method of scheduling. En lieu of (or in addition to) try
syntax, you can now check in a file called 'try_task_config.json' to the root of the source
tree. The format is either a list of task labels, or dict where task labels are the keys.
Taskcluster will simply schedule any tasks that are listed there.
This file is primarily meant to be generated by tools (which don't exist yet), as the json
format is much easier for tools to generate or consume. These tools should use an in-memory
commit to add the file so it is automatically removed again after the push.
A server-side hook will be added in bug 1380357 to prevent this file from accidentally
landing on non-try trees.
MozReview-Commit-ID: 2zKfZXuuDhH
--HG--
extra : rebase_source : b5d5ff47c607288657418fd041603093f8c29e85
The toolchain jobs produce artifacts that are going to be used by other
jobs, but there is no reliable way for the decision task to know the
name of those artifacts. So we make their definition required in the
toolchain job definitions.
--HG--
extra : rebase_source : eb668cf2bf7793b44dc3d91e3ddb8d7c69e3de52
This name can be used by user interfaces to find the action corresponding to a
dedicaetd UI element.
MozReview-Commit-ID: HFBDlX30J31
--HG--
extra : rebase_source : a70c66106f71f46fb4678eb076f53db073fc83f3
Minor modificatinons based on conversations at the SF all-hands.
MozReview-Commit-ID: AnkwXppUPJd
--HG--
extra : rebase_source : b3bef018689a614dd6a3048e85db30bb4f5722e5
Support OSX Signed nightlies (in the complete.mar too)
MozReview-Commit-ID: HkGRRm7k2Ra
--HG--
extra : rebase_source : a8fc5c76bdc623e4217840b7b75c39d0aa0b9051
Support OSX Signed nightlies (in the complete.mar too)
MozReview-Commit-ID: HVJwMuXUixX
--HG--
extra : rebase_source : b05bca1a539e751e8a8a59c32a0a392752c95c73
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: BeXoChssNjF
--HG--
extra : rebase_source : 1f086e96411a7683b77cfecb6079b54ab9b0f643
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: 2zOkiBS294y
--HG--
extra : rebase_source : d380d4975f8480b65d44c07c099ef6f166228e01
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: BeXoChssNjF
--HG--
extra : rebase_source : 1f086e96411a7683b77cfecb6079b54ab9b0f643
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: 2zOkiBS294y
--HG--
extra : rebase_source : d380d4975f8480b65d44c07c099ef6f166228e01
{central, autoland, inbound} are logically treated as a single unit for
many tasks and policies. Let's formalize that collection via a "trunk"
alias.
MozReview-Commit-ID: H4JPTyu2J2F
--HG--
extra : rebase_source : f4cabfc48e24b9f55b833bd95bdbf81b036cee6d
The purpose of this parameter has been superseded by the `include_nightly`
property.
MozReview-Commit-ID: 4iXQsv9Drqg
--HG--
extra : rebase_source : c94142282909a88c29fe6809d87721bef1f198c2
This is a more robust approach than using substring matching on task labels.
As an optimization, this simply avoids creating balrog tasks for unsigned beets
using only-for-attributes, rather than omitting them in a transform.
MozReview-Commit-ID: 8MNOxu0WgXo
--HG--
extra : source : 1aeb99ce3e6c2576b7b9b71c1cdf97a1d1889a49
This is a more robust approach than using substring matching on task labels.
As an optimization, this simply avoids creating balrog tasks for unsigned beets
using only-for-attributes, rather than omitting them in a transform.
MozReview-Commit-ID: 8MNOxu0WgXo
--HG--
extra : rebase_source : 9e93a996241bcb0d345c18813919a41320e95651
