Having both .*-qr/.* and windows.* as patterns in the run-on-projects
test platform list means that a platform like windows10-64-qr matches
both and the parser complains. Instead of doing this, we can remove the
windows patterns and take the talos tests out of the windows-talos
test-set which accomplishes the same thing.
MozReview-Commit-ID: 9kMooiNiHTb
--HG--
extra : rebase_source : e58d74c43531cb1f4b625b188be696d1e951eda0
With run-by-manifest, the debug reftests timeout on OSX. Increasing
the chunks by 1 seems to improve them.
MozReview-Commit-ID: 14xidhwXCue
--HG--
extra : rebase_source : b3042c3a65c67fc54af5d64624427593e48c8364
The webgl mochitests will run on all nightly branches (so inbound, autoland,
m-c, try) by default.
MozReview-Commit-ID: E3KPTbbxE3E
--HG--
extra : rebase_source : 4001221ba39b5b7f00dc70c8cd2b1e3a18599cd7
The Proguard dependency is now managed by Gradle.
MozReview-Commit-ID: EOvKSE5z28P
--HG--
extra : rebase_source : 760b117f500cc639cc8c24e9c02933990f358dd7
- add balrog submit-toplevel - this replaces the final portion of the updates builder.
- rename balrog transform to balrog_submit, because it's for balrog locale submission
- make this default to the 'promote' phase. balrog and beetmover currently take the current
phase, which isn't always the wanted behavior.
- rename balrog publish to balrog schedule
- add balrog secondary push and secondary scheduling, for RCs
- remove the release_updates transforms
- make the task.py balrog transforms smarter
- get rid of the release_balrog_publishing transforms; ad a generic worker_type transform
- add BALROG_ACTIONS to scriptworker.py
- add get_balrog_action_scope()
- remove the unused balrog channel scopes
MozReview-Commit-ID: 369ACiOAd5F
--HG--
rename : taskcluster/taskgraph/transforms/balrog.py => taskcluster/taskgraph/transforms/balrog_submit.py
extra : rebase_source : 9311522460ae6790af14a6b8b9600019702f8cbd
We now have access to workers running on EC2 instances with dozens of
vCPUs. gecko-<L>-b-linux-large is m4.10xlarge, m5.12xlarge, c5.9xlarge,
or c4.8xlarge. gecko-<L>-b-linux-xlarge is m5.24xlarge, m4.16xlarge,
or c5.18xlarge.
Experimentation reveals that Clang tasks are the only tasks that
are CPU efficient enough (read: cost effective) to run on these
larger worker types.
This commit defines the new worker types and switches Clang toolchain
tasks to run on the new workers. clang5 and clang6 tasks take ~30 minutes
on the -large variant but ~17 minutes on the -xlarge variant. All other
tasks don't show as linear of a speedup. So running them on the
-xlarge variant isn't justified.
As part of this change, Mac toolchain tasks have been converted
to run on gecko-<L>-b-linux* workers. The gecko-<L>-b-macosx64 workers
are actually Linux. IMO the b-macosx64 worker type is no longer needed.
Moving the toolchain tasks off the worker should hopefully not be very
controversial.
MozReview-Commit-ID: HynQPMWiWHo
--HG--
extra : rebase_source : 1142767e2a51c17880909ec6f15b694db8a43af2
Virtual function declarations should specify only one of `virtual`, `final`, or `override`, as per the Mozilla C++ style guide:
https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Coding_Style
This lint warns about:
virtual void Bad1() final
void Bad2() final override
void Bad3() override final
Caveats: This lint doesn't warn about `virtual void NotBad() override` at this time because there are 8000+ instances. It also doesn't warn about function declarations that span multiple lines because the regex can't match across line breaks.
MozReview-Commit-ID: LcBsOAKKgz7
--HG--
extra : rebase_source : 4da72ffac59acdc9796e3f540f24bb97af989cd0
Update sccache build description to use the latest stable
rust toolchain. We didn't upgrade earlier because of problems
on Windows.
Note that we can't just depend on the stable rust toolchain
alias, because the toolchain deps are resolved in a single
pass. Instead, we use the current stable version explicitly.
MozReview-Commit-ID: 4OVbFsYZZLZ
--HG--
extra : rebase_source : 5b65d05f646f061a4018e0fad2e31e48e884912a
The upload is handled by taskcluster, which is handled by copying files around,
so remove support for specifying alternative commands.
Differential Revision: https://phabricator.services.mozilla.com/D540
--HG--
extra : rebase_source : 968249c0a308178b62af77d48e6aa307d4192e5a
Run unit tests under geckoview/ when running 'mach android test'. This
also lets us run those tests on Taskcluster.
The test report parser for 'mach android test' had a bug where the input
directory was wrong. As a result, we weren't producing test output at
all. This patch fixes the input directory, and outputs an error if no
reports are found at all to avoid this bug in the future.
MozReview-Commit-ID: IiswQaSPCr0
Also add rc-{google-play-track,rollout-percentage} for RC pushapk.
One nice side effect of using the same push-apk kind: we don't re-run push-apk during the ship_fennec relpro flavor if we've run the ship_fennec_rc flavor with the same build. (Google Play would reject the same buildid.)
This is really for bug 1433536, but MozReview is forcing me to include this patch with the others for reasons.
MozReview-Commit-ID: 69ymqVLv9E2
--HG--
extra : rebase_source : b87d4dd2394788a5452ff3f52a8ca5022a15b9ee
extra : intermediate-source : 7a826d274a4828018a836cf1149df29d403a7c11
extra : source : a3c60b0370a3e08ce765f87c1d7d5dad24879881
We'd like to install the NDK through the Android SDK manager. But we
can't pin versions of the NDK with the SDK manager, and so Google
can silently upgrade the NDK on us. Since that is undesirable, this is
the next best thing.
With the toolchain task in hand, we can make all the relevant tasks
depend on the toolchain task and remove the download of the NDK from
tooltool as well.
Android non-gradle tests run against an Android apk built without
gradle, "Bng" on treeherder; confusingly, this platform is labelled
"Android API16+ Gradle opt". This patch significantly reduces the
tests run on this platform: now only robocop tests are run.
The full set of Android tests, including mochitests and reftests,
continue to run on the "Android 4.3 API16+ opt" platform.
We don't actually go install the package, but if a one-click loaner user
goes on to apt-get install gdb, they will get a version that is useful,
rather than the version in wheezy that won't give them e.g. variables
information.
--HG--
extra : rebase_source : 1f1ba607a759fc3136c59513773a043e8e8680c0
The GDB version in Debian wheezy doesn't handle the DWARF data that the
GCC version we use to build Firefox and toolchains produce. So we take
the GDB version from Debian stretch and backport it.
--HG--
extra : rebase_source : dae0e9dcd5dde5a7c74b6cefd560480fccd9c5fa
We start from the image used for Firefox builds, and add the debug
packages for all the system libraries.
--HG--
extra : rebase_source : 2c759975d9837beabdc08a15fd926a99fd1cecf8
The binutils we currently use as part of our GCC toolchain artifact
doesn't understand some relocations in the CRT objects on Debian
stretch, making the embedded CRT objects from bug 1427344, which we want
to remove in bug 1431251, necessary.
OTOH, there is no benefit from using our GCC toolchain artifact for host
compilations on those builds. In fact, Android builds, which are in a
similar position, being built on Debian stretch and being cross-builds,
don't care to use our GCC toolchain artifact.
It's arguably a good thing that those builds are not tied to the version
of GCC we use to build Firefox for linux, so let's remove this
dependency.
--HG--
extra : rebase_source : a80d4e4fb01a4862b844ebde0c521a635f462c0a
Don't build ucl when building upx, Debian stretch has a recent enough
version. In fact, the last upstream version doesn't build with GCC in
Debian stretch (http://bugs.debian.org/811707)
--HG--
extra : rebase_source : aae67773b9dd3b99f6ddf9ab7f59a628037e6925
This job requires cmake, which should be fixed, but in the meanwhile,
create a separate docker image with it installed, based on the image we
use for other spidermonkey builds.
--HG--
extra : rebase_source : da43a7999b6bd86dbba816358d907c902415bed4
That image is used to derive all the debian7-* images, and its
definition is parametrized, which will allow to create other images
based on other versions of Debian, from the same definition.
XZ_OPT is kept in each of those because we don't want to automatically
set it in all further derived images.
--HG--
extra : rebase_source : 7f4597c1ea4af83627a9373dbdc7945d20b7d996
It's been more than two weeks since the 1.23 stable release, and
we're making official builds with that toolchain release, so begin
requiring that version so new language features can be used in
development.
MozReview-Commit-ID: E6WuP41ceTn
--HG--
extra : rebase_source : 75850dd9edbf8e3f9beab394e4af7fad76ce3b17
New upstream release.
- Avoiding argument copies improves memory footprint.
- RwLock<T> no longer requires T to be Send.
- AsciiExt trait methods are now directly available
on str, [u8], u8, and char types without a `use`
statement.
MozReview-Commit-ID: 7Rx8uoNTMqH
--HG--
extra : rebase_source : 39d6297d61d19d710a1376557e4b4d81bdab02c9
This patch disables the stdcxx-compat check for the sm-fuzzing build which
requires patching autospider as well. Furthermore, it switches the build
to linux64-clang-6-pre because the older clang 3.9 does not support trace-pc
instrumentation. Finally, it excludes fuzzing parts from the vanilla allocation
check.
MozReview-Commit-ID: FdhCIFdUore
--HG--
extra : rebase_source : c41bda01cb42f2ef0cd5a1675d88bdb55d9dc8c9
Switch almost all builds currently using the desktop-build image to use
the right debian7-*-build image instead. The only exception is the rust
bindgen spidermonkey builds, that require cmake being installed, but I
don't want to add it to the base images because that involves risking a
cmake dependency unwantedly slipping in Firefox (rust-bindgen ironically
requires cmake to build a single C++ file...)
It is no longer used as part of the build, now that publishing is handled in
separate tasks.
Differential Revision: https://phabricator.services.mozilla.com/D373
--HG--
extra : rebase_source : a6e31fe5d554076d4da584f0993c21b6c1a8ebd0
This adds an 'override' for the default scheduling component for tests, which
is based on their suite.
MozReview-Commit-ID: 6vd8sb2zeuU
--HG--
extra : rebase_source : 25988c6790287e01fa7751effa72e8b924858948
There were a few constraints in the choice of the version of dpkg to
backport:
- 1.17.20 is the first version that supports the debian source format
for that xz-utils package.
- versions >= 1.17.10 and <= 1.17.22 fail to build on wheezy.
- versions >= 1.17.21 depend on a version of patch not available on
wheezy.
All in all, the simpler choice was to go with version 1.17.20 with a
backport of the build failure fix.
That version of dpkg breaks the version of devscripts in wheezy, so the
version from wheezy-backports would be better to use, but we can't
unconditionally use it on all builds, because it happens that
mk-build-deps from that version is broken with the dpkg in wheezy.
In the end, it's simpler to build that backport and rely on package task
dependencies rather than selectively install the package from
wheezy-backports, so we do that. Except we can't use version
2.14.11~bpo70+1 because of bug 1419577.
--HG--
extra : rebase_source : 19ad1a44b770229fbc7e15bbcf01d3cb101315a8
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 978cf033732cbbbb277d206dec69660175b82afa
Back in bug 1360609, we added `run-on-projects` to a list so that the
toolchain tasks wouldn't run on every push on release branches.
Fast forward to now, and they're depended upon by other tasks, meaning
they are triggered when appropriate, without resorting to that trick. In
fact, the commit message for bug 1360609 said we could switch to an
empty list once the jobs have dependencies.
The same is true from package tasks, which, in fact, I suspect would
happen on every push on release branches.
The only exception is for a few toolchains that are depended upon by
nothing, and that are produced for developer consumption with e.g. mach
artifact toolchain.
--HG--
extra : rebase_source : bb8624fed7490b85f4bd72b7ceb2db7a72b4c2ab
There are now only a handful of buildbot jobs remaining and the concern over
outdated treeherder exclusion profiles has largely been resolved.
This does remove the tc() group from a substantial number of tasks which will
now show up as top level tasks, potentially adding clutter. In some cases, we
might want to re-add a new group (e.g group builds or compiled tests together).
However rather than try to predict the best group names for tasks I'm unfamiliar
with, I think it's best to land this as is. Then if things are looking too
cluttered at the root namespace, file follow-up bugs as needed.
MozReview-Commit-ID: 8SMwjDwAOzV
--HG--
extra : rebase_source : 2f6d89d11c139bdcd404e7537db799d0e36ee4c3
Add a geckoview-docs job that executes "./mach android geckoview-docs",
which takes care of calling gradle to generate the javadoc archive, and
uploading it to Github using given parameters.
MozReview-Commit-ID: DTWh4XdFZEO
--HG--
extra : rebase_source : 9d75be24cb553b3a773d3d34a2bdbdf4d4c8cd34
With the use of job-defaults, we can avoid a lot of repetition from
those definitions.
--HG--
extra : rebase_source : 932c2ed530aa8aec9a33da60cf652535fa0bd303
The mozharness transform is supposed to set the docker image to
desktop-build when not already set, but was not doing it properly.
I guess this is why some jobs were setting the image themselves, despite
using the mozharness transform.
Consequently, don't manually set the image to desktop-build when it's
the default.
--HG--
extra : rebase_source : 024bd10960bedaee3416785348a5c12498c5286f
At the same time, restrict the installed packages to the script
requirements to build Firefox. Toolchains have their own image so we
don't need to install packages for them.
--HG--
extra : rebase_source : c0e7aa178b1ce2ceb01f9dfe6af37bbb54d4d708
The one available in Debian wheezy is 1.7.10.4, which is really old, and
on our centos images, we're using 2.8.0rc3, which, while old too, is
more modern. While we may want to go with a more recent version, I'd
rather avoid differing from what we currently use, so use the exact same
version.
--HG--
extra : rebase_source : dfdf75a635073c248faef8a67648b2a83e4a1d84
... except libdmg-hfsplus. RedHat decided to patch libbz2 to have a
different soname, so a binary built on Debian can't run on
RedHat/CentOS. Ironically, a binary built on RedHat/CentOS can run
on Debian. While we could use some tricks to make libdmg-hfsplus built
on Debian work, at this point, it's not worth the effort. We can live
with libdmg-hfsplus being built on CentOS until the builds that use it
switch to Debian, which is imminent.
... and except mingw32-nsis. Sourceforce renewed their certificate last
week and somehow the corresponding CA is not yet recognized by the
ca-certificates in Debian wheezy (an update is underway but see below)
... and except wine, because it requires more 32-bits packages than can
be installed on the toolchain-build docker image. But all things
considered, the mingw32 builds don't need to be using the same docker
images as the linux builds, and they could be, like the android builds,
be based on a more recent build image. So the corresponding toolchains
can be built on a more recent version of Debian too.
Consequently, we keep all the mingw32 related toolchains on the
desktop-build image for now.
It was failing to build with the GCC/binutils on the CentOS-based docker
image, but it doesn't with the Debian-based one, so we can remove the
dependency on the gcc toolchain task. This allows sccache to remain
untouched when we change the gcc build scripts, and more importantly,
this allows it to depend on no toolchain that requires building things.
This makes it now possible to use sccache as a dependency for all other
toolchains jobs that compile, if that's beneficial (which might not be
the case, given the current sccache retention time, but at least it's a
viable option, now)
This marks **/docs/** as exclusively docs, and code that is autodoc'd as
inclusively docs.
That means that a change that purely modifies documentation files will *only*
run `docs` tasks, while a change that modifies autodoc'd source code will
*additionaly* run `docs` tasks. The tasks do not run by default.
MozReview-Commit-ID: G9tOK0AwtrI
--HG--
extra : rebase_source : 8dd971e5c9b0eb5f47895664a4ea49442f303ecb
extra : source : 0881de9b2b5e36ec37cc866f1d4af109da57a919
This makes use of pytest's generation feature. To add a new
template test, just add a new entry containing the input and
expected output to the dict in test_templates.py
MozReview-Commit-ID: 4qMefYHMjAp
--HG--
extra : rebase_source : ba3049885d1a2485048e1ff9913be43317559376
This is a minor cleanup of the python.yml source test tasks.
MozReview-Commit-ID: 6UanmbZHF8P
--HG--
extra : rebase_source : e06d310af9ca05bfdab1bc1e3bd2bc6aa3035cb9
This marks **/docs/** as exclusively docs, and code that is autodoc'd as
inclusively docs.
That means that a change that purely modifies documentation files will *only*
run `docs` tasks, while a change that modifies autodoc'd source code will
*additionaly* run `docs` tasks. The tasks do not run by default.
MozReview-Commit-ID: G9tOK0AwtrI
--HG--
extra : rebase_source : 8dd971e5c9b0eb5f47895664a4ea49442f303ecb
extra : source : 0881de9b2b5e36ec37cc866f1d4af109da57a919
We've had problems with crashes in llvm-dsymutil for a while, and while
they are, in essence, due to the fact that rustc produces bad debug
info, they are a hurdle to our builds. The tool comes along clang, and
updating clang is not necessarily easy (witness bug 1409265), so, so
far, we've relied on backporting fixes, which can be time confusing
(witness bug 1410148).
OTOH, llvm-dsymutil is a rather specific tool, that doesn't strictly
need to be tied to clang. It's only tied to it because it uses the llvm
code to do some of the things it does, and it's part of the llvm source
tree. But it could just as well be a separate tool, like it was(is?) on
OSX.
So, we add a toolchain job to build it from the llvm source,
independently from clang, so that we can update it separately, if we
hit new crashes that happen to already be fixed on llvm trunk. It will
also allow to more easily update after upstream fixes crashes after we
report them.
--HG--
extra : rebase_source : b814353b4b4632e46646a24b8f54c5300618ff49
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 73e8fc51ea53af1e647fc1d5093c67d614dd009e
The one available in Debian wheezy is 3.81, but we're explicitly using
4.0 on CentOS, most notably because of its --output-sync option which
helps make logs better in some ways.
This takes the package from Debian jessie and builds it for Debian
wheezy.
--HG--
extra : rebase_source : 20bb550703fec41ed0175ef7f78c5b9a394160f3
This marks **/docs/** as exclusively docs, and code that is autodoc'd as
inclusively docs.
That means that a change that purely modifies documentation files will *only*
run `docs` tasks, while a change that modifies autodoc'd source code will
*additionaly* run `docs` tasks. The tasks do not run by default.
MozReview-Commit-ID: G9tOK0AwtrI
--HG--
extra : rebase_source : 8dd971e5c9b0eb5f47895664a4ea49442f303ecb
extra : source : 0881de9b2b5e36ec37cc866f1d4af109da57a919
