This change switches most CI builds to clang, with a few exceptions:
- valgrind builds, until bug 1481670 is figured out.
- PGO and nightly builds, until that's fully tested.
- coverage builds, per bug 1471339 comment 17.
- base toolchain builds, to keep some builds on GCC even when we're
fully switched to clang.
- any build that doesn't use build/unix/mozconfig.linux (e.g. probably
all those driven by autospider.py, maybe others).
This change switches most CI builds to clang, with a few exceptions:
- valgrind builds, until bug 1481670 is figured out.
- PGO and nightly builds, until that's fully tested.
- coverage builds, per bug 1471339 comment 17.
- base toolchain builds, to keep some builds on GCC even when we're
fully switched to clang.
- any build that doesn't use build/unix/mozconfig.linux (e.g. probably
all those driven by autospider.py, maybe others).
While some builds have a PERFHERDER_EXTRA_OPTIONS environment set on the
taskcluster side, many others have the equivalent set at the mozharness
level. But only the former are actually linted against, which,
unsurprisingly, translates to conflicting values between some of the
mozharness configs.
So we move those configurations to taskcluster, enable the lint on all
the kinds that look like builds (based on them using the build_attrs
transform), and adjust the values to stop conflicting. Notably, for
searchfox and static-analysis-autotest.
--HG--
extra : rebase_source : 097333608e61e1df66e5d8f914e15784f35e58f2
There are now only a handful of buildbot jobs remaining and the concern over
outdated treeherder exclusion profiles has largely been resolved.
This does remove the tc() group from a substantial number of tasks which will
now show up as top level tasks, potentially adding clutter. In some cases, we
might want to re-add a new group (e.g group builds or compiled tests together).
However rather than try to predict the best group names for tasks I'm unfamiliar
with, I think it's best to land this as is. Then if things are looking too
cluttered at the root namespace, file follow-up bugs as needed.
MozReview-Commit-ID: 8SMwjDwAOzV
--HG--
extra : rebase_source : 2f6d89d11c139bdcd404e7537db799d0e36ee4c3
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : d8447b5422e63e88444008fddb76d658829694de
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : 0aa9ef8151c2fccf62507dfecc0bc57b157772e1
Add a toolchain job description which calls the
repack_rust.py script to package the requested
upstream build of Rust and its standard libraries
for use in gecko builds.
Links are added to these new toolchains for various build
and analysis tasks as appropriate. The base-toolchain
tasks use an explicitly-versioned toolchain since those
can be different from the current release used for most builds.
The corresponding tooltool manifest entries are removed
now that taskcluster artifact versions are available.
This simplifies the update process since new toolchains
can be packaged and used automatically by just updating
the versions in the task descriptions.
A 'linux64-rust' toolchain can be added to other tasks
as a dependency and artifact. It supports linux64-
hosted builds of Rust code targeting linux64 or linux32.
A 'linux64-rust-macos' toolchain targets linux64-hosted
builds of Rust code targeting macOS on x86_64.
A 'linux64-rust-android' toolchain targets linux64-hosted
builds of Rust code targeting various Android architectures.
Two 'win64-rust' and 'win32-rust' toolchain tasks create
similar entries for Windows-hosted builds. All our automation
builds are hosted on win64, so we could use one artifact
with support for both targets, but currently this doesn't
work because of cross-compilation issues in some crates.
This patch maintains the previous separation between
win32 and win64 rust toolchains until that can be addressed.
MozReview-Commit-ID: GRiJml8CtzO
--HG--
extra : rebase_source : 09a3698ce7f9a8b5f2b5d9b5a1fde9c05dc6b540
bug 1382280 tripled the size of desktop-build image because of
installation of debug symbols. It is only used for valgrind, so let's
move valgrind task to use its own image.
MozReview-Commit-ID: 16St7dDj8tr
--HG--
rename : taskcluster/docker/desktop-build/Dockerfile => taskcluster/docker/valgrind-build/Dockerfile
extra : rebase_source : cc66813cab430d906643fbadf63c661e14784f6f
After bug 1388569 and bug 1388572, all jobs that have needs-sccache set
have a dependency on either linux64-sccache or win64-sccache, and
vice-versa. Which means they are now redundant, and one should imply the
other.
--HG--
extra : rebase_source : ae72f67ccf2da7ba645416b8be4d10687005d01a
This leaves out fuzzing and linux static analysis builds, which are
using, respectively, clang 4.0.1 and clang 3.8, while linux64-clang
produces a 3.9 and win*-clang a 5.0
--HG--
extra : rebase_source : 45128ac74bf4fe7e6a2ace57043c34ecdf0fe929
Such a definition automatically sets up the corresponding dependencies
in the taskgraph, and adds the necessary artifact definitions for use in
the corresponding jobs. The jobs end up with a MOZ_TOOLCHAINS
environment variable with a list of path@task-id strings, where task-id
is corresponding to the (possibly optimized) toolchain job, and path
corresponding to the toolchain-artifact defined for that toolchain job.
--HG--
extra : rebase_source : b2d297bd75d9c416b30d2a6c6d61efcb64681727
Since bug 1321847, mozharness tooltool manifests can be overriden from
the environment. We use that possibility to now define tooltool
manifests from taskcluster job definitions. Ideally, we'd also remove
the definitions from the mozharness configs, but with things still
running on buildbot, it's not clear what things might break because of
that. We'll do it in a separately back-out-able followup.
--HG--
extra : rebase_source : 860b8f1d4fdc4a557770a3749055f19b1ec45e93
Previously, mozharness defined a separate action to collect build
metrics. This required the script and/or config to define that
action.
Metrics collection for CI is important. So it should be enabled by
default.
This commit changes the "build" action/method to always call the
metrics collection function after successful build. References to
the "generate-build-stats" action have been removed because it is
redundant.
A side-effect of this change is we may generate build metrics where
we weren't before. This could lead to e.g. duplicate entries in some
Perfherder series. Let's see what breaks ;)
MozReview-Commit-ID: 42UQI5YQTMC
--HG--
extra : rebase_source : c57dc9ec6ac46003384edff098a0ad81c75539b7
extra : source : c9812dd7d27a174c0ee46d44ec595fbe29c9e1db
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
--HG--
extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
The "generate-build-stats" mozharness action collects a bunch of build
system metrics, including build_resources.json, ctors count, and
installer size and reports them by writing special messages that
log parsers read.
Before this commit, this mozharness action occurred sometime after
"build." But relative ordering to other actions was not consistent
and appears to be significantly cargo culted.
4e61e69a383c (bug 1304508) changed the "check" mozharness action to
invoke `mach build check` instead of `make` directly. An unintended
consequence of this is that `mach build` replaced the
build_resources.json file from the build itself with one measuring
`make check`. This made the "build time summary" metric take a nose
dive.
This commit works around the issue introduced by 4e61e69a383c by
reordering the mozharness actions so "generate-build-stats" follows
immediately after the "build" action. Not only does it now occur
before the "check" action, but it also occurs before uploading and
other actions.
I'm not sure why "generate-build-stats" is its own action and not
part of "build" itself. As the diff shows, numerous instances of
"generate-build-stats" are commented out, which means we aren't
collecting metrics. "generate-build-stats" is also missing from a
handful of mozharness configs, including Windows Clang builds. I'm
not sure what the history is (it is likely varied and almost certainly
involves a fair amount of cargo culting), but I think it is a bug that
we aren't collecting build metrics for every build. I would like to
fix this. And moving "generate-build-stats" immediately after "build"
should make that transition easier.
MozReview-Commit-ID: 7jNTVWRvMnh
--HG--
extra : rebase_source : 0b5fd1f462caa5c283ba7e1b693fdc5b8b948add
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
