Add a toolchain job description which calls the
repack_rust.py script to package the requested
upstream build of Rust and its standard libraries
for use in gecko builds.
Links are added to these new toolchains for various build
and analysis tasks as appropriate. The base-toolchain
tasks use an explicitly-versioned toolchain since those
can be different from the current release used for most builds.
The corresponding tooltool manifest entries are removed
now that taskcluster artifact versions are available.
This simplifies the update process since new toolchains
can be packaged and used automatically by just updating
the versions in the task descriptions.
A 'linux64-rust' toolchain can be added to other tasks
as a dependency and artifact. It supports linux64-
hosted builds of Rust code targeting linux64 or linux32.
A 'linux64-rust-macos' toolchain targets linux64-hosted
builds of Rust code targeting macOS on x86_64.
A 'linux64-rust-android' toolchain targets linux64-hosted
builds of Rust code targeting various Android architectures.
Two 'win64-rust' and 'win32-rust' toolchain tasks create
similar entries for Windows-hosted builds. All our automation
builds are hosted on win64, so we could use one artifact
with support for both targets, but currently this doesn't
work because of cross-compilation issues in some crates.
This patch maintains the previous separation between
win32 and win64 rust toolchains until that can be addressed.
MozReview-Commit-ID: GRiJml8CtzO
--HG--
extra : rebase_source : 09a3698ce7f9a8b5f2b5d9b5a1fde9c05dc6b540
Those resources are used to compute a unique identifier for the
toolchain, and changes to those files will change the unique identifier,
and lead to the toolchain being rebuilt.
Using wildcards, especially in the build-clang directory, makes all the
files from there used for the unique identifier, even files irrelevant.
The side effect is that any change to any json file for clang toolchains
currently triggers *all* clang toolchains to be rebuilt, which is a
waste of resources and time.
But while it is tempting to list all the files involved, it is also
tedious and error-prone. Specifically, listing the relevant patch files
for clang toolchain builds is bound to end up outdated. OTOH, we're not
trying to mitigate bad actors here, but just to avoid shooting ourselves
in the foot. And patch files are, in practice, not changed. The jsons
are changed to reference them or not, but the patches themselves don't
change in relevant ways. They may be updated for new versions of clang,
which require a json change anyways. So we ignore the patch files.
None of the toolchain jobs are actually using sccache, so they don't
need the dependency. Those jobs also don't run frequently enough that
sccache would be a win for them, and when they do run, they're likely
to have few cache hits, if the cache is not expired anyways.
--HG--
extra : rebase_source : bddd0d0688303f9d9b117f4e2b952f22621bf9dd
The MinGit tooltool package used for Windows builds comes straight from
https://github.com/git-for-windows/git/releases/
This builds the version currently used on automation.
--HG--
extra : rebase_source : dbc2a36b07611e673d6661032ad53123a688d422
The toolchain jobs produce artifacts that are going to be used by other
jobs, but there is no reliable way for the decision task to know the
name of those artifacts. So we make their definition required in the
toolchain job definitions.
--HG--
extra : rebase_source : eb668cf2bf7793b44dc3d91e3ddb8d7c69e3de52
The toolchain build scripts are currently defining the tooltool
manifest they use on their own. We move the definitions to the
taskcluster job definitions to normalize on everything using that.
--HG--
extra : rebase_source : cbab2e32d78d711fcb595763d0c2600c7c0c423e
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
--HG--
extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
We have a `unittest_try_name` for all unit test tasks, and similarly a
`talos_try_name`. The `-j` flag controls tasks known as "jobs" (although the
word has dozens of other meanings, too). Some of those set `job_try_name`, but
others do not and have special-case support in `try_option_syntax.py`. With
this change, all "jobs" set `job_try_name` and the special-case is removed.
MozReview-Commit-ID: 9hvW7wBIl2B
--HG--
extra : rebase_source : 120b5e9e7aa8f81fe49e72f4dadafdbd145ac357
This allows to find them and optimize them out during the taskgraph
optimization phase, and will allow to get toolchain artifacts through a
mach command for developers.
The index path is generated similarly to git trees or mercurial
manifests, and allows to find the right task corresponding to the the
contents of the files in the task `extra.resources` along the toolchain
scripts.
`when.files-changed` is not used when a task has index paths because we
need tasks to happen independently of whether there were changes to
those files when the index or artifacts expire.
--HG--
extra : rebase_source : e9995cee0ee39d7b64090a243e380aeae336a69f
The toolchain tasks are hard to spot on treeherder, in the ocean of
build and test jobs associated with the platforms they are currently
under.
Now that we have a significant number of toolchain tasks across
different platforms, it's even worse, especially combined with the fact
that they don't happen on every push.
To make them more easily visible, we move them to a new, separate,
"platform", with the name "toolchains", instead of having them in
different platforms. But since the distinction between Linux, OSX and
Windows 32/64 is still interesting to have, we create groups for each of
those platforms.
But because of bug 1215587, the jobs still end up associated to their
previous group, defeating the new grouping, so to work around that bug,
we also rename the jobs in subtle ways.
--HG--
extra : rebase_source : 6c093c070c18a64eba1c21bf2a2c97b2a9aaabc5
This seems like something that the in-tree taskcluster infrastructure
ought to be able to handle, but in the absence of that functionality,
this is the next best option.
The build-clang-windows.sh is basically an exact copy of the Linux one
at the moment; we'll fix it up for all the Windows-specific taskcluster
bits in a future commit.
