It can be necessary to use the artifacts from package tasks as build
dependencies for other package tasks.
--HG--
extra : rebase_source : a625670ae7af9a99c848924bdba9368c556e6766
Not all changes to debian_packages.py lead to actual changes to the
corresponding tasks. And since the tasks are pretty much self-contained,
we can consider that variations of them will be entirely represented in
the command used for the task. The only exception is the patch content
when there is an associated patch.
--HG--
extra : rebase_source : e70fa41a824106b5ceca09fd535c0c36bb0739ac
In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : 8c76bc91e1d5102f68c43e1050d61971fef32e9f
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 978cf033732cbbbb277d206dec69660175b82afa
Back in bug 1360609, we added `run-on-projects` to a list so that the
toolchain tasks wouldn't run on every push on release branches.
Fast forward to now, and they're depended upon by other tasks, meaning
they are triggered when appropriate, without resorting to that trick. In
fact, the commit message for bug 1360609 said we could switch to an
empty list once the jobs have dependencies.
The same is true from package tasks, which, in fact, I suspect would
happen on every push on release branches.
The only exception is for a few toolchains that are depended upon by
nothing, and that are produced for developer consumption with e.g. mach
artifact toolchain.
--HG--
extra : rebase_source : bb8624fed7490b85f4bd72b7ceb2db7a72b4c2ab
There are now only a handful of buildbot jobs remaining and the concern over
outdated treeherder exclusion profiles has largely been resolved.
This does remove the tc() group from a substantial number of tasks which will
now show up as top level tasks, potentially adding clutter. In some cases, we
might want to re-add a new group (e.g group builds or compiled tests together).
However rather than try to predict the best group names for tasks I'm unfamiliar
with, I think it's best to land this as is. Then if things are looking too
cluttered at the root namespace, file follow-up bugs as needed.
MozReview-Commit-ID: 8SMwjDwAOzV
--HG--
extra : rebase_source : 2f6d89d11c139bdcd404e7537db799d0e36ee4c3
When trying to remove an ubiquitous group like tc(), it's hard to tell where the
error was located without grepping my filesystem. This makes it a bit easier to
find and fix these errors.
MozReview-Commit-ID: 8NjvB5zOoqb
--HG--
extra : rebase_source : 167d3097f96548cf9c13b602d7d485cb69d48c2d
Relying on the various transforms setting it manually is error prone,
and, in fact, is why bug 1430037 busted beta. This change makes this
setting happen at a single place. This yields the same full task graph
as before, except for *more* chain-of-trust inputs being set now: they
were missing for toolchain tasks (which makes us closer to bug 1384430).
--HG--
extra : rebase_source : b6bf3a3b6da7174957c4c6814b853a51ee8a1e27
Add a geckoview-docs job that executes "./mach android geckoview-docs",
which takes care of calling gradle to generate the javadoc archive, and
uploading it to Github using given parameters.
MozReview-Commit-ID: DTWh4XdFZEO
--HG--
extra : rebase_source : 9d75be24cb553b3a773d3d34a2bdbdf4d4c8cd34
With the use of job-defaults, we can avoid a lot of repetition from
those definitions.
--HG--
extra : rebase_source : 932c2ed530aa8aec9a33da60cf652535fa0bd303
The mozharness transform is supposed to set the docker image to
desktop-build when not already set, but was not doing it properly.
I guess this is why some jobs were setting the image themselves, despite
using the mozharness transform.
Consequently, don't manually set the image to desktop-build when it's
the default.
--HG--
extra : rebase_source : 024bd10960bedaee3416785348a5c12498c5286f
At the same time, restrict the installed packages to the script
requirements to build Firefox. Toolchains have their own image so we
don't need to install packages for them.
--HG--
extra : rebase_source : c0e7aa178b1ce2ceb01f9dfe6af37bbb54d4d708
The one available in Debian wheezy is 1.7.10.4, which is really old, and
on our centos images, we're using 2.8.0rc3, which, while old too, is
more modern. While we may want to go with a more recent version, I'd
rather avoid differing from what we currently use, so use the exact same
version.
--HG--
extra : rebase_source : dfdf75a635073c248faef8a67648b2a83e4a1d84
... except libdmg-hfsplus. RedHat decided to patch libbz2 to have a
different soname, so a binary built on Debian can't run on
RedHat/CentOS. Ironically, a binary built on RedHat/CentOS can run
on Debian. While we could use some tricks to make libdmg-hfsplus built
on Debian work, at this point, it's not worth the effort. We can live
with libdmg-hfsplus being built on CentOS until the builds that use it
switch to Debian, which is imminent.
... and except mingw32-nsis. Sourceforce renewed their certificate last
week and somehow the corresponding CA is not yet recognized by the
ca-certificates in Debian wheezy (an update is underway but see below)
... and except wine, because it requires more 32-bits packages than can
be installed on the toolchain-build docker image. But all things
considered, the mingw32 builds don't need to be using the same docker
images as the linux builds, and they could be, like the android builds,
be based on a more recent build image. So the corresponding toolchains
can be built on a more recent version of Debian too.
Consequently, we keep all the mingw32 related toolchains on the
desktop-build image for now.
Because Ubuntu 16.04 changed in a way that busts gl3 tests and we can't
update the desktop1604-test image anymore.
--HG--
extra : amend_source : bfa07f9f77990dd6915b8c92d218227436bc6fc4
The install-mercurial.sh script currently installs a global mercurial
configuration after installing mercurial manually. In order to share
that configuration with docker images installing a mercurial package
through packages tasks, we move it to a separate file.
We however keep the part setting web.cacerts in install-mercurial.sh,
since it uses a path depending on what kind of environment the script is
run. Moreover, the instructions that come with mercurial to build
Debian packages come with web.cacerts set to the right path already, so
it's not needed in that case anyways.
At the same time, use multiple files in /etc/mercurial/hgrc.d/ instead
of a single /etc/mercurial/hgrc file.
--HG--
extra : rebase_source : 8140d8243cf012489025afe058f467c72224c891
Build dependencies won't be installed from backports unless they are not
satisfiable in the given Debian release. This is useful to get dh-python
on Wheezy.
--HG--
extra : rebase_source : 1f249b4ceae4fdd9ea37e9a9b9e9b62b48a1c9ed
In the case of mercurial, we don't want to use a .dsc as the original
source, but rather use the debian packaging scripts available in the
upstream mercurial tarball.
--HG--
extra : rebase_source : ec5b288f3994bc0bc1ec9ebce40def807bb7681f
The taskgraph.util.schema.check_schema function validates key names used
in schemas, ensuring they are dashed lower-case. However, it currently
assumes keys are either direct strings, Required or Optional entries,
and either ignores or fails to recognize other patterns.
For example, it ignores Any, and fails to recognize combinations like
Required(Any(...)), which we're going to use in next patch.
--HG--
extra : rebase_source : 4f6ff51a4a9dc9c7d9b6d070e03c6cc6e1befe80
I don't intend to install ccache in the Debian build images. Hazard
builds are the only builds running on the desktop-build image still
using ccache somehow, and that gains them nothing, since the ccache
directory is not set to a cached directory on taskcluster, meaning
the build always starts with an empty cache. If anything, this currently
makes the build slower.
Eventually, those builds should be able to use sccache, once the
necessary setup moves out of mozconfig.cache.
--HG--
extra : rebase_source : fba6dab78b25ea61892cbe6127ead36da395b0e0
taskcluster/taskgraph/transforms/task.py sets an expiry to 28 days for
tasks on try, vs. 1 year on other projects, but only do that when an
expiry is not already set, which docker images do. And they do always
set to 1 year.
But it doesn't make sense to keep large docker images from try for a
year. So use the same retention policy as the default one. We /could/
just remove the expiry from docker images and get the task.py default,
but it seems like whatever future change might happen to that default
shouldn't affect docker images, so it's better to duplicate the setting.
--HG--
extra : rebase_source : b5b46ca34a40ac82c5403b67d5b1aacf8cf8cceb
It was failing to build with the GCC/binutils on the CentOS-based docker
image, but it doesn't with the Debian-based one, so we can remove the
dependency on the gcc toolchain task. This allows sccache to remain
untouched when we change the gcc build scripts, and more importantly,
this allows it to depend on no toolchain that requires building things.
This makes it now possible to use sccache as a dependency for all other
toolchains jobs that compile, if that's beneficial (which might not be
the case, given the current sccache retention time, but at least it's a
viable option, now)
New Android-Gradle plugins pin the build-tools version, and we want to
be consistent between Gradle and moz.build.
MozReview-Commit-ID: ApWS4rHzPuH
--HG--
extra : rebase_source : 38a9781c472d858f3300cbbcbdc6d2311c465713
Turns out Google's maven repository doesn't publish checksums. I
can't imagine why not, but there it is. We have to think more about
whether to trust the artifacts downloaded from maven.google.com.
MozReview-Commit-ID: CdWijorq1IV
--HG--
extra : rebase_source : a884971e51ce7b1ff993754b130f462c476646ab
New Android-Gradle plugins pin the build-tools version, and we want to
be consistent between Gradle and moz.build.
MozReview-Commit-ID: ApWS4rHzPuH
--HG--
extra : rebase_source : 5a5730b4b9ce84af40a7c73c4f1abba017103f02
Turns out Google's maven repository doesn't publish checksums. I
can't imagine why not, but there it is. We have to think more about
whether to trust the artifacts downloaded from maven.google.com.
MozReview-Commit-ID: CdWijorq1IV
--HG--
extra : rebase_source : e4373273e7aea7df79d70b5fbc233233a84d2360
This marks **/docs/** as exclusively docs, and code that is autodoc'd as
inclusively docs.
That means that a change that purely modifies documentation files will *only*
run `docs` tasks, while a change that modifies autodoc'd source code will
*additionaly* run `docs` tasks. The tasks do not run by default.
MozReview-Commit-ID: G9tOK0AwtrI
--HG--
extra : rebase_source : 8dd971e5c9b0eb5f47895664a4ea49442f303ecb
extra : source : 0881de9b2b5e36ec37cc866f1d4af109da57a919
This makes use of pytest's generation feature. To add a new
template test, just add a new entry containing the input and
expected output to the dict in test_templates.py
MozReview-Commit-ID: 4qMefYHMjAp
--HG--
extra : rebase_source : ba3049885d1a2485048e1ff9913be43317559376
This is a minor cleanup of the python.yml source test tasks.
MozReview-Commit-ID: 6UanmbZHF8P
--HG--
extra : rebase_source : e06d310af9ca05bfdab1bc1e3bd2bc6aa3035cb9
This marks **/docs/** as exclusively docs, and code that is autodoc'd as
inclusively docs.
That means that a change that purely modifies documentation files will *only*
run `docs` tasks, while a change that modifies autodoc'd source code will
*additionaly* run `docs` tasks. The tasks do not run by default.
MozReview-Commit-ID: G9tOK0AwtrI
--HG--
extra : rebase_source : 8dd971e5c9b0eb5f47895664a4ea49442f303ecb
extra : source : 0881de9b2b5e36ec37cc866f1d4af109da57a919
And trigger a new Ubuntu 16.04 docker image with a different hash to get
the latest version of the GL drivers that were updated to llvmpipe 5
recently. Those new drivers make Firefox enabled
WEBGL_compressed_texture_s3tc, making the test pass.
--HG--
extra : rebase_source : 09a8995829f985aef29a8919fecaefaf90791584
We've had problems with crashes in llvm-dsymutil for a while, and while
they are, in essence, due to the fact that rustc produces bad debug
info, they are a hurdle to our builds. The tool comes along clang, and
updating clang is not necessarily easy (witness bug 1409265), so, so
far, we've relied on backporting fixes, which can be time confusing
(witness bug 1410148).
OTOH, llvm-dsymutil is a rather specific tool, that doesn't strictly
need to be tied to clang. It's only tied to it because it uses the llvm
code to do some of the things it does, and it's part of the llvm source
tree. But it could just as well be a separate tool, like it was(is?) on
OSX.
So, we add a toolchain job to build it from the llvm source,
independently from clang, so that we can update it separately, if we
hit new crashes that happen to already be fixed on llvm trunk. It will
also allow to more easily update after upstream fixes crashes after we
report them.
--HG--
extra : rebase_source : b814353b4b4632e46646a24b8f54c5300618ff49
In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : bbe8b001849e59bb655bb0e9766a6071ad38a52c
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 73e8fc51ea53af1e647fc1d5093c67d614dd009e
The one available in Debian wheezy is 3.81, but we're explicitly using
4.0 on CentOS, most notably because of its --output-sync option which
helps make logs better in some ways.
This takes the package from Debian jessie and builds it for Debian
wheezy.
--HG--
extra : rebase_source : 20bb550703fec41ed0175ef7f78c5b9a394160f3
This marks **/docs/** as exclusively docs, and code that is autodoc'd as
inclusively docs.
That means that a change that purely modifies documentation files will *only*
run `docs` tasks, while a change that modifies autodoc'd source code will
*additionaly* run `docs` tasks. The tasks do not run by default.
MozReview-Commit-ID: G9tOK0AwtrI
--HG--
extra : rebase_source : 8dd971e5c9b0eb5f47895664a4ea49442f303ecb
extra : source : 0881de9b2b5e36ec37cc866f1d4af109da57a919
This boils down to always setting worker.env to avoid a KeyError.
MozReview-Commit-ID: 1s4az9BFcc2
--HG--
extra : rebase_source : dabed2dedb00d176b829c6c0ff911e0236c5dec4
When looking for perfherder data collection duplicates, we currently
keep full job objects references, which are then used in case an error
occurs, to display the job names of the duplicates.
But those job objects are yielded and may be modified by other
transforms, and presently, by the time a duplicate is found, the
corresponding job object has been modified such that it has no 'name'
key anymore, leading to a KeyError exception when trying to display
the duplicate error message.
So instead of keeping the job objects, which can change, and which we
don't have a real use for, just keep the job name.
--HG--
extra : rebase_source : 204e90a6fe1e4ce62f361451e1176d3195a3383b
In bug 1427326, we added package tasks that can be depended upon by
docker image tasks. In that case, we add the routes containing a digest
for those package tasks when computing the docker image digests.
The problem is that those routes start with 'index.gecko.cache.level-n'
where n varies between try and e.g. mozilla-central. That means the
digest for those docker images varies between try and e.g.
mozilla-central, which then prevents try from using the cached versions
for mozilla-central when there is one, like for other docker images
without package dependencies.
What we really need from those routes is the digest part, which is
independent of the level, and we don't actually care about anything else
in the route string, so just use the digest.
--HG--
extra : rebase_source : 4aecf8472306963da34f2bd4d92675962c0432bc
This was useful when we still had buildbot-based build jobs, but all
it achieves nowadays is add friction when adding new build jobs on
taskcluster.
--HG--
extra : rebase_source : aa6a21a875eff1888c16900acf6d01ff37ab832b
New upstream release.
- Avoiding argument copies improves memory footprint.
- RwLock<T> no longer requires T to be Send.
- AsciiExt trait methods are now directly available
on str, [u8], u8, and char types without a `use`
statement.
MozReview-Commit-ID: 7Rx8uoNTMqH
--HG--
extra : rebase_source : 54068e34eaf6ccdbcc854fafb94d2a66fd068adf
There are e.g. some build infrastructure changes that we want to have a
controlled impact on the Firefox builds we produce. We have, in multiple
occasions, gone through manual work to compare Firefox builds, most of
the time using the diffoscope tool (https://diffoscope.org/).
This change introduces a new task kind that takes two Firefox builds as
input, either by name (reference to a build from the current task graph)
or by index (reference to a build from a previous push), and compares
them.
In order to get a Firefox build by index, we rely on dummy tasks with
an optimization we expect to always hit, so we add the necessary bits
to ensure those dummy tasks can go through up to the optimization phase
and be optimized out there.
--HG--
extra : rebase_source : 37482f67652dab2fcef2db4e6b8efe653999bae5
The spidermonkey mozjs and rust-bindings builds run sed on
$topsrcdir/.cargo/config.in to generate the cargo config they use, but
they previously only replaced the @top_srcdir@ substitution. This patch
makes them replace any other substitutions with an empty value to add
a bit of future-proofing.
MozReview-Commit-ID: 1DzP9vXxHMD
--HG--
extra : rebase_source : e8c0268a2a6e91ca2000b340beee2dcff0636591
We currently use a 32-bit Rust toolchain for win32 builds, but this can lead
to OOM situations. This patch makes win32 builds use a 64-bit Rust toolchain,
which requires a little bit of extra configuration because rustc needs to
be able to find a link.exe that produces 64-bit binaries for building
things like build scripts, which are host binaries.
We will now generate a batch file that sets LIB to the paths to 64-bit
libraries and invokes the x64-targeting link.exe, and add a section to the
.cargo/config file to instruct cargo to use that batch file as the linker
when producing 64-bit binaries.
MozReview-Commit-ID: 9vKBbm7Gvra
--HG--
extra : rebase_source : 599b3b661c7a8a5db1f32a2a9732fc202fb55e1e
Add a cross-compilation copy of rust's standard library targeting
i686-pc-windows-msvc to the win64-rust toolchain package so it
can be used to build for win32 as well.
MozReview-Commit-ID: 3598VZrDjIH
--HG--
extra : rebase_source : f1b25a68a67ae7f9c505a42d17f29dbedf59a49d
Instead of duplicating Dockerfiles between taskcluster/docker/*
directories, which can be error prone for very close images, it can be
desirable to use the same file. This change allows to set the
`definition` keyword on a docker image definition in kind.yml that
will make the task use the files from taskcluster/docker/<definition>
instead of taskcluster/docker/<image_name>.
--HG--
extra : rebase_source : 11ae231f66ca6a77896c1cff6c1580d04210f052
Ideally, we'd simply use the --build-arg docker argument along with ARG
in the Dockerfile, but that's only supported from Docker API 1.21, and
we're stuck on 1.18 for the moment.
So we add another hack to how we handle the Dockerfile, by adding a
commented syntax that allows to declare arguments to the Dockerfile.
The arguments can be defined in the docker images kind.yml file through
the `args` keyword. Under the hood, they are passed down to the docker
image task through the environment. The mach taskcluster-build-image
command then uses the corresponding values from the environment to
generate a "preprocessed" Dockerfile for its context.
--HG--
extra : rebase_source : 26a43dd680c1ab97b1a4689a23c55594a3b21b67
Landings of e.g. bug 1427336 triggered new toolchain jobs. One of those
jobs, because of wrong changes in bug 1421100, downloaded a new rust
compiler beta instead of the intended fixed beta version. In turn, that
new rust compiler beta fails to compile the slog crate.
Now, because of how toolchain cache indexes work, every new win32 job
picks that new unintended rust compiler beta version, even on branches
where 1427336 hasn't landed.
I couldn't find a way to force the right beta version, so we're pretty
much stuck with that toolchain index pointing to the wrong version of
rust beta.
By backing out bug 1426324, we return to a toolchain index that is known
to have an artifact for the right rust compiler beta.
Unfortunately, if something triggers a new TW32(rust) job after this,
that toolchain index will be busted as well.
Giving a directory to %include would copy all leaf files under one
single directory in the context image. The only image affected is
valgrind-build, which ended up having a dot-config/pip.conf file instead
of dot-config/pip/pip.conf, meaning valgrind jobs weren't using the
pip config.
--HG--
extra : rebase_source : 518c8ca1617b57ae4b4bb83a85340de5515c26c5
Android cppunit, test-verify, and mochitest-gpu have been running on lower end
aws instances. It is probably better to run all emulator tasks on xlarge.
libcrypto, part of OpenSSL, and that dmg links against, has a varying
ABI, and something built against libcrypto on Centos won't run on Debian
and vice versa. It might not even work between versions of the same OS
(e.g. Debian 7 vs. Debian 9).
Because of that, it is desirable to statically link it.
This incorporates https://github.com/mozilla/libdmg-hfsplus/pull/1
and sets OPENSSL_USE_STATIC_LIBS when building libdmg-hfsplus.
--HG--
extra : rebase_source : 21a46f707494f388a899c08d0923f8b393d12cd1
fontconfig uses expat by default to read its xml configuration, but when
expat is not there at build time, it falls back to libxml2. We ended up
in this situation, while on Debian, fontconfig is built against expat.
This makes no practical difference, since we're not actually using
fontconfig, but for some reason, the difference in dependencies has an
impact on how ld chooses to arrange the .plt and .got.plt sections,
meaning that even though the code and data is originally identical, in
the .o files, the resulting linked machine code is largely different
because of all the applied relocations changing the offsets of e.g. call
instructions for function calls through the .plt. This results in large
differences in .plt, .init, .text, .data.rel.ro, as well as symbols list
when building on Debian.
This thus is meant to help make the differences more tractable.
--HG--
extra : rebase_source : 7a731c34074a50e84412f73ab9499248345fb14a
OSX (cross) repackages are currently using a tooltool manifest to get
libdmg and hfsplus. Change those jobs to use the toolchain artifacts
instead.
At the same time, modify the repackage mozharness script's _run_tooltool
so that it doesn't fail with MOZ_TOOLCHAINS being set but without a
tooltool_manifest_src, matching the similar function in buildbase.py.
--HG--
extra : rebase_source : d128d4709c5d1d28d1a6b9c585fde82e99f725c7
I suppose it was setup through ~worker/.hgrc before we started
installing a /etc/mercurial/hgrc that enables a few other extensions
and sets some preferences.
There is no reason to now have two places where mercurial is being set
up, and it feels natural that we set it up at the system level.
Ideally, we'd also clean up the centos6-based images, but they require
an update of the centos6-build and centos6-build-upd images on the
docker hub, which is not really convenient, and those images are going
to be obsoleted soon anyways (bug 1399679).
--HG--
extra : rebase_source : 32c9cdf5d0fe8ac2c60a1c5a38e572c83a4783b2
Bug 1389715 removed the image definition in taskcluster/ci/docker-image
as well as the files associated with it in
taskcluster/docker/desktop-test, but the Dockerfile in there was the
only use of the Ubuntu 12.04 setup script, so it is currently unused.
--HG--
extra : rebase_source : 7d8018e7c94e2625ff9822a2d66231722a030394
The debian-build-system-setup.sh script doesn't use the install-make and
install-cmake scripts, so it's unnecessary to install them in /setup.
--HG--
extra : rebase_source : 4ba24b9827e67b9c7ad203e789e00e19d37786da
The script was added in bug 1179893 but looks like it has actually never
been used. A duplicate of the file was used for the upload-symbols
image, but that was removed in bug 1422740.
Since it was the only file in desktop-build/bin, we stop copying the
directory in docker images.
--HG--
extra : rebase_source : 4bcdb5ba0118e87455c6f596bf54e4528fe1b1ef
While we're here, add a missing prepare_vcs_checkout for the
comm-central checkout.
--HG--
extra : rebase_source : 788a288330e34b5551ec2b12726a755e268566c2
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : d8447b5422e63e88444008fddb76d658829694de
We're about to remove some tooltool manifests, so we need those calls to
work properly when TOOLTOOL_MANIFEST is not set.
--HG--
extra : rebase_source : 89d41021a87915dc9133e61543352e3bda1dace4
Back when we started needing gtk+3 to build Firefox, we were using mock
to setup the build environment, and a tooltool package was the most
sensible way to handle this.
Fast forward to today, and we're close to moving the build environment
to Debian, which comes with gtk+3 packages. But in order to simplify
the various checks for the transition, it is desirable to stop using the
tooltool package. Which we can actually do in a reasonable way now that
we use docker images instead of mock, by building and installing gtk+3
in the build environment images.
So we modify the script that was producing the gtk+3 tooltool packages
such that it installs gtk+3 in the docker images, both 32 and 64 bits.
And invoke it when creating the desktop build environment docker images.
--HG--
extra : rebase_source : 75e987d6de7f3ae8a3d9b478fc173e191d28aace
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : 0aa9ef8151c2fccf62507dfecc0bc57b157772e1
We're about to remove some tooltool manifests, so we need those calls to
work properly when TOOLTOOL_MANIFEST is not set.
--HG--
extra : rebase_source : 38ca0e3b894097ed3667901b05af79062a6c82c2
Back when we started needing gtk+3 to build Firefox, we were using mock
to setup the build environment, and a tooltool package was the most
sensible way to handle this.
Fast forward to today, and we're close to moving the build environment
to Debian, which comes with gtk+3 packages. But in order to simplify
the various checks for the transition, it is desirable to stop using the
tooltool package. Which we can actually do in a reasonable way now that
we use docker images instead of mock, by building and installing gtk+3
in the build environment images.
So we modify the script that was producing the gtk+3 tooltool packages
such that it installs gtk+3 in the docker images, both 32 and 64 bits.
And invoke it when creating the desktop build environment docker images.
--HG--
extra : rebase_source : fe18bfb2ec8db183c44838d5a7a0051322b2a9c0
On Debian, the autoconf binary is autoconf2.13 while it is autoconf-2.13
on Centos.
In make-source-package.sh, we need to run autoconf to generate the
old-configure to include in the package, so try both.
In hazard-analysis.sh, we actually don't need autoconf itself, so just
copy configure.in to configure.
--HG--
extra : rebase_source : d21075394c69cd7cd6738da645173eb29f4a1259
Note that we need to use the virtual-with-gpu instances on windows for
WebRender to even start up.
MozReview-Commit-ID: 6fMDun7casP
--HG--
extra : rebase_source : 5068bd17d11725c2c0f5bd0b387a54047475f0c6
For example, jittests will be an inclusive test suite; all files which might
affect the suite are tagged with
SCHEDULES.inclusive += ['jittest']
but those files usually also schedule all of the exclusive components (including
the platform families android, linux, macosx, and windows). This makes sense:
those files could potentially affect any other test suite on any platform too.
But the jittest job on Android, for example, needs to run only if the jittest
component is scheduled -- it does not need to just because something
Android-related changed. So its optimization should be {skip-unless-schedules:
['jittest']}, not {skip-unless-schedules: ['jittest', 'android']}.
This fix "figures out" the distinction by looking at what kind of component the
test suite is. Maybe that is too magic, and we should also have to write
"component: implicit" in the tests/*.yml file.
MozReview-Commit-ID: EIsVvi1vziE
--HG--
extra : rebase_source : eb7ad26db801028dc514af6c2eaaadb649445db0
We build packages of the same versions that were installed by
taskcluster/docker/recipes/install-cmake.sh and
taskcluster/docker/centos6-build/system-setup.sh in the desktop-build
image.
--HG--
extra : rebase_source : 843b89065daabd450f54ebf7a2cf55d00977e23a
This also renames the existing test sets for qr to be linux-specific, so
we can have a different test set for windows QR builds. The windows10-64-qr
gpu mochitests will run on all nightly branches (so inbound, autoland, m-c,
try) by default.
MozReview-Commit-ID: F2NjCTHYg13
--HG--
extra : rebase_source : eb107b11d995a84bd76885e1af241ca05f634684
Previously we had linux64-qr as the only QuantumRender test platform.
Soon we will have windows10-64-qr as well and (eventually) we will have
some macOS -qr tests as well. So this patch generifies the existing
regexes to match these platforms.
In a couple of places redundant platform matching lines were removed, to
avoid the case where a given platform (e.g. windows10-64-qr) matches
multiple regexes (e.g. .*-qr/.* and windows.*) which produces an error.
MozReview-Commit-ID: 8YO9lQETVYM
--HG--
extra : rebase_source : 60b59fedd7cab71f7cf2118feea16b058bd4654c
We currently use a 32-bit Rust toolchain for win32 builds, but this can lead
to OOM situations. This patch makes win32 builds use a 64-bit Rust toolchain,
which requires a little bit of extra configuration because rustc needs to
be able to find a link.exe that produces 64-bit binaries for building
things like build scripts, which are host binaries.
We will now generate a batch file that sets LIB to the paths to 64-bit
libraries and invokes the x64-targeting link.exe, and add a section to the
.cargo/config file to instruct cargo to use that batch file as the linker
when producing 64-bit binaries.
MozReview-Commit-ID: 9vKBbm7Gvra
--HG--
extra : rebase_source : 366dd966cafe4f07b8e59fc170d2db2dada32627
Add a cross-compilation copy of rust's standard library targeting
i686-pc-windows-msvc to the win64-rust toolchain package so it
can be used to build for win32 as well.
MozReview-Commit-ID: 3598VZrDjIH
--HG--
extra : rebase_source : baab6d8718d7a8d38a353a2bffcea14dcee45c8f
The "contract" for toolchains is that extracting foo.tar.xz creates a
directory named foo/. That is however not true for mingw32.tar.xz, which
extracts into gcc/, possibly overwriting files from the gcc.tar.xz
archive (which is also used for mingw builds, for the host part).
This is also not true for nsis.tar.xz, but it reportedly has problems
when it's not in the same directory as mingw32.
But mingw32 doesn't actually need to be mixed with gcc, so it's better
to separate them as they are supposed to be.
--HG--
extra : rebase_source : 30d90af64459bbb31bc076e48f3c661fa9cd4a79
f179a112278d (bug 1373878) established tasks for Rust tests. It
created new Treeherder "platforms" for each task.
These platforms (which still only have a single task) seem wasteful.
Let's remove the one-off Treeherder platform and move the "rusttests"
tasks into an existing platform so Treeherder's output is more
concise.
MozReview-Commit-ID: 8Fcph0r5wad
--HG--
extra : rebase_source : 3035d0ea50208911440498a108f653c298903352
The symbol-upload task currently downloads the symbols-full.zip artifact
from the build task and then uploads it to the symbol server. These zip
files can be very large (>1GB) so we spend a lot of time doing that.
Now that we're uploading to Tecken instead of Socorro, we can instead
just send the URL of the artifact to Tecken's upload API and ask it to
fetch that directly:
https://tecken.readthedocs.io/en/latest/upload.html#upload-by-download-url
This should make the symbol upload task a fair bit faster.
MozReview-Commit-ID: 8HcbgrWYT1O
--HG--
extra : rebase_source : 4e8f7a28c956befb3e291e8be4d41a2b6728e5cd
Building for some tier-3 platforms imply building without a JIT, and it
happens quite regularly that this setup is broken by API changes in
Spidermonkey.
This adds a new job with JIT disabled, but skip tests for now because
some fail or crash.
--HG--
extra : rebase_source : 3c6e1dfb3cd7d0bff59c494f6230c9f1b55479ed
Adjust post-beetmover-dummy's tasks and deps.
- Fennec doesn't have beetmover-repackage or beetmover-checksums,
so add `beetmover` to the post-beetmover-dummy kind-dependencies.
- Add a fennec-promote post-beetmover-dummy job.
- Remove the extraneous -ship post-beetmover-dummy jobs. Once we
removed the assumption that dummy jobs had to be in the same phase,
these became redundant.
In testing, this looks good. For the next step, we may want to split
these dummy tasks up by `build_platform`. Then downstream tasks could
then optionally filter their dummy deps by `build_platform`; this
would allow for certain platforms to proceed on to the next steps
sooner, rather than wait for the slowest platform to finish.
I also suspect we don't need post-beetmover-checksums-dummy at all;
it's redundant.
MozReview-Commit-ID: EeHjwTQnVB1
--HG--
extra : rebase_source : 812288cf083499d38e3e47a203c43163afd8e2a5
extra : source : e78626133e88e124922a43b5af7ebfd5e5325360
`taskgraph.create` can add an additional dependency to tasks prior to task submission. The queue will error out if we submit a task with over 100 dependencies, so we should limit ourselves to 99 dependencies here.
MozReview-Commit-ID: ClT0vjYBPp4
--HG--
extra : rebase_source : e1f168a5c472be8d45e689517fff0a47ba1bbe7c
extra : source : 351a176ad1181d9a43056098a66b80f1fa56e401
..and remove support for when.files-changed in the test kind. It is still used
for other kinds, and that will be addressed in other bugs.
This is re-landing of this bug, now without running test-verify excessively.
MozReview-Commit-ID: GBilXAktICZ
--HG--
extra : rebase_source : 6cc9a3b5a365d74689946bfa0296f51bc08c2113
libcrypto, part of OpenSSL, and that dmg links against, has a varying
ABI, and something built against libcrypto on Centos won't run on Debian
and vice versa. It might not even work between versions of the same OS
(e.g. Debian 7 vs. Debian 9).
Because of that, it is desirable to statically link it.
This incorporates https://github.com/mozilla/libdmg-hfsplus/pull/1
and sets OPENSSL_USE_STATIC_LIBS when building libdmg-hfsplus.
--HG--
extra : source : 78f2064b3811db58b364c32ce9b58a3f2dcaf8f8
There is no /lib64 on Debian. OTOH, one doesn't need to give the full
path to a system library in LDFLAGS, so just use -l syntax instead.
--HG--
extra : rebase_source : b795f97ab209499824afa5ef1aee9da52657ceb9
also add devedition and the missing linux{,64} balrog_props.json (by
defining taskcluster_nightly configs for devedition).
MozReview-Commit-ID: 3MAYjSL20FV
--HG--
extra : rebase_source : 45c7d6e63c18f77d9434cebeb65d05608f7d2508
extra : histedit_source : 13b4d908a02571f6c8506fe27de2edfe7342f424
This allows us to funnel large numbers of tasks down to avoid hitting
MAX_DEPENDENCIES. I avoided using a morph here because we might break
certain cot assumptions.
MozReview-Commit-ID: BIILM9O6CI4
--HG--
extra : rebase_source : 48bd11e8b6f25887671aafec23b2a27aad98b9d1
extra : histedit_source : 7bd193e12043272ed4ea6059260ed7abfca4d1d1
Of note, we have a get_phase() method that lets us determine the phase from the target_tasks_method.
MozReview-Commit-ID: LE7PLbMX3oU
--HG--
extra : rebase_source : 8db7e55963473bc9f11b47dd40b41339cb2bac1b
extra : histedit_source : 99f1eefacb43ce9d0177fc87e328e5eeb43b6b35
Here we're adding/updating support for the promote/push/ship phases
for fennec, devedition, and firefox. These are now keyed off of the new
shipping_phase and shipping_product attributes as much as possible.
MozReview-Commit-ID: Fkg8jTPeZHZ
--HG--
extra : rebase_source : 7a31959e410baa812c12177bc56e48c05b523b6b
extra : histedit_source : d88468da698cca83ea16a2f309dccfd4a569b171
In bug 749312, we were given permission to create a source readme
instead of a source tarball. This will save us cycles, disk, and
human configuration time.
We still need to address the missing balrog_props.json for
beetmover-source for that task to turn green.
MozReview-Commit-ID: wnyPoNXCsH
--HG--
extra : rebase_source : 843751523e1fce5743849f43796788dbba5115d3
extra : histedit_source : 2993eb186dc7bd71ad35af48d4393803b0b147dc
