I tested this on automation and the build went on, though I couldn't test
the bindgen output because the build right now is busted on one dependent crate
with rust beta, which is the first toolchain that has this package, and will go
to release shortly.
This should work though! If I need more changes I'll adjust them in bug 1432153.
You can test the repackage manually with repack_rust.py --toolchain beta, for
example.
MozReview-Commit-ID: GI2f6vGVqTe
Don't build ucl when building upx, Debian stretch has a recent enough
version. In fact, the last upstream version doesn't build with GCC in
Debian stretch (http://bugs.debian.org/811707)
--HG--
extra : rebase_source : aae67773b9dd3b99f6ddf9ab7f59a628037e6925
This job requires cmake, which should be fixed, but in the meanwhile,
create a separate docker image with it installed, based on the image we
use for other spidermonkey builds.
--HG--
extra : rebase_source : da43a7999b6bd86dbba816358d907c902415bed4
We've observed apt failures multiple times where it apparently fails to
get a file in full from snapshot.debian.org. Making it retry
automatically rather than retriggering tasks seems better.
--HG--
extra : rebase_source : f3ffb415ccc30b7e7c44e6a48b29eb20e69efdd5
The apt in Debian stretch doesn't allow repositories with a Release file
not being GPG signed. Setting up GPG signatures on the
taskcluster-artifact-based repositories is a tricky process, and not
strictly necessary. It turns out not creating a Release file at all
works just as well, and works across all current Debian versions. The
packages priorities remain the same, such that packages from those
repositories are still prefered over the ones from the main Debian
repository (as long as versions are higher).
See comment in cloud-mirror-workaround.sh for that part.
--HG--
extra : rebase_source : df5af330859a314285a6c1922d899489997d2f19
That image is used to derive all the debian7-* images, and its
definition is parametrized, which will allow to create other images
based on other versions of Debian, from the same definition.
XZ_OPT is kept in each of those because we don't want to automatically
set it in all further derived images.
--HG--
extra : rebase_source : 7f4597c1ea4af83627a9373dbdc7945d20b7d996
The base images from docker hub actually contain a
/etc/apt/apt.conf.d/docker-clean that does the equivalent of an apt-get
clean after installing packages.
--HG--
extra : rebase_source : 190de9e3b10a0309cf9cfb3260a91477a5a93ba3
python-dev was required to build mercurial, but the need for that was
removed in bug 1429669.
The others were required for mingw32 toolchains, but they are using a
different docker image and will switch to another different docker image.
--HG--
extra : rebase_source : b65c586a325f220c565e79afb3d3c9acc9f922bc
So far, the best we've been able to do is to upload an image to the
docker hub, and point an image's Dockerfile's FROM to the version
uploaded onto the hub.
That is a cumbersome process, and makes the use of "layered" docker
images painful.
This change allows to declare a parent docker image in the
taskcluster/ci/docker-image/kind.yml definitions, which will be
automatically loaded before building the image. The Dockerfile can then
reference the image, using the DOCKER_IMAGE_PARENT argument, which will
contain the full image name:tag.
Some details are left off, for now, such as VOLUMEs. At this point,
VOLUMEs should all be defined in leaf docker images.
--HG--
extra : rebase_source : 221cff0ca5a91d694ff5c3626fe707c15ba45e23
Now that `mach taskcluster-build-image` can, we can avoid all the manual
handling based on curl and jq in the image builder.
An additional advantage on relying on `mach taskcluster-build-image`
doing more is that less changes to the build-image.sh script will be
necessary, and thus less updates of the image builder docker image.
--HG--
extra : rebase_source : dd174d60675e41e4391894f28235c674c1840829
This allows to avoid writing out a tar file to then extract it to feed
it to `docker build`. This is essentially what the image-builder docker
image does, except it uses a temporary file for the tar.
--HG--
extra : rebase_source : 8275d737e02714fc198d3ba3d3e62e3f18d8e0bf
While spawning `docker load` is likely to work on developer machines,
on automation, it requires a docker client that is the exact same
version as the server running on the taskcluster worker for
docker-in-docker, which is not convenient. The API required for `docker
load` is rather simple, though, and can be mimicked quite easily.
While this change in itself is not necessary for developer machines,
it will allow to re-use the same command for the image-builder to
load a parent docker images when deriving one from another. We could
keep a code branch using `docker load` but it seems wasteful to maintain
two branches when one can work for both use cases.
--HG--
extra : rebase_source : d72956d7dd329b92564cbaa3fbfe0687d4d5d994
The zstd command we spawn, if available at all, might be the wrong
version: zstd changed its stream format in an incompatible way at some
point, and the version shipped in e.g. Ubuntu 16.04 uses the old format,
while the version taskcluster relies on uses the new format.
Relying on gps's zstandard library allows to ensure we use the right
version. Another advantage is that we can trivially pip install it in a
virtualenv if it isn't available on the system running the command.
If we're ridding ourselves of the subprocess spawning for zstd, we might
as well cover curl as well. Especially considering the error handling
when subprocesses are involved is not trivial, such that the current
error handling code is actually broken and leads to dead-lock
conditions, when, for example, curl is still waiting for the python side
to read data, but the python side is not reading data anymore because
an exception was thrown in the tar reading loop.
--HG--
extra : rebase_source : 054c37cfaa68bf475b37545ebaa99144584b93d4
The used pattern:
except Exception:
error = sys.exc_info()[0]
finally:
...
if error:
raise error
actually loses everything that is interesting about the original
exception. Not catching the exception just makes it thrown up the stack,
except when a different exception is thrown from the finally block,
which is what that if error: raise error is attempting to do... except
it doesn't throw the original exception, but its type only.
--HG--
extra : rebase_source : 17601fcc90fcdfefd93c4267f3cd33425d5326fd
Now that we don't need to read the contents of a file to hash the
contents of a docker image context, we can avoid creating a file
in generate_context_hash.
--HG--
extra : rebase_source : 98abe9bfdc48b612a3d251296991d0f769b449fd
This will allow us, down the line, to avoid creating a file at all in
some cases.
--HG--
extra : rebase_source : e4ea40341836cf24aa6d61c905b2efa660ee13f2
It's been more than two weeks since the 1.23 stable release, and
we're making official builds with that toolchain release, so begin
requiring that version so new language features can be used in
development.
MozReview-Commit-ID: E6WuP41ceTn
--HG--
extra : rebase_source : 75850dd9edbf8e3f9beab394e4af7fad76ce3b17
When we branched to beta, the MinGW build started failing with a strange error
like "could not find the robustcheckout Mercurial extension". We disabled it
for a while, but now we're re-enabling it (and fixing the problem.)
The root culprit of this was that we were using the incorrect mozconfig. MinGW
does all sorts of stuff in the mozconfig, but the beta branch overrides the
mozconfig using platform_overrides in
testing/mozharness/configs/builds/branch_specifics.py
We avoid this override by changing the MinGW platform so it doesn't match
and the mozconfig doesn't get overridden.
MozReview-Commit-ID: JkETWCRHacO
--HG--
extra : rebase_source : 5ef7d96804acd3a4eaa6dd6cc8b483f3de438b61
New Android-Gradle plugins pin the build-tools version, and we want to
be consistent between Gradle and moz.build.
MozReview-Commit-ID: ApWS4rHzPuH
--HG--
extra : rebase_source : 22008e9333b15c594ce26c2a52f67396d6e3ab84
extra : source : f918500d9cf5112b70bc8e0a120df435b02252b7
Turns out Google's maven repository doesn't publish checksums. I
can't imagine why not, but there it is. We have to think more about
whether to trust the artifacts downloaded from maven.google.com.
MozReview-Commit-ID: CdWijorq1IV
--HG--
extra : rebase_source : 6c66cf1444876624f10409ea6437863e2c2ea9b0
extra : source : 0850b319efd43ac8f4d61485451722975da55ca1
New upstream release.
- Avoiding argument copies improves memory footprint.
- RwLock<T> no longer requires T to be Send.
- AsciiExt trait methods are now directly available
on str, [u8], u8, and char types without a `use`
statement.
MozReview-Commit-ID: 7Rx8uoNTMqH
--HG--
extra : rebase_source : 39d6297d61d19d710a1376557e4b4d81bdab02c9
This patch disables the stdcxx-compat check for the sm-fuzzing build which
requires patching autospider as well. Furthermore, it switches the build
to linux64-clang-6-pre because the older clang 3.9 does not support trace-pc
instrumentation. Finally, it excludes fuzzing parts from the vanilla allocation
check.
MozReview-Commit-ID: FdhCIFdUore
--HG--
extra : rebase_source : c41bda01cb42f2ef0cd5a1675d88bdb55d9dc8c9
So we can change a string in this file to force the use of new caches,
thus ensuring a clean break from one changeset to the next.
MozReview-Commit-ID: EZsR23a1PZE
--HG--
extra : amend_source : bf6a3c53ab14f2eb249fdf57ff70dce97388f0a0
Switch almost all builds currently using the desktop-build image to use
the right debian7-*-build image instead. The only exception is the rust
bindgen spidermonkey builds, that require cmake being installed, but I
don't want to add it to the base images because that involves risking a
cmake dependency unwantedly slipping in Firefox (rust-bindgen ironically
requires cmake to build a single C++ file...)
Bump mingw version to get the newest commit and do not include the
un-needed dw-extras.h on MinGW (thanks Jacek!)
MozReview-Commit-ID: OjO93XHCxs
--HG--
extra : rebase_source : 933bbb385004988a23d1069c9cd3241b3a3b336e
It is no longer used as part of the build, now that publishing is handled in
separate tasks.
Differential Revision: https://phabricator.services.mozilla.com/D373
--HG--
extra : rebase_source : a6e31fe5d554076d4da584f0993c21b6c1a8ebd0
This makes the XZ_OPT environment variable set for every task that runs
on those docker images, effectively enabling parallel compression for
all of them.
For e.g. the gcc 4.9 toolchain build task, this makes the final
compression step take 23s instead of > 3 minutes.
--HG--
extra : rebase_source : 83809f1f72007cc0829573f1def29a9afdc300c9
This adds an 'override' for the default scheduling component for tests, which
is based on their suite.
MozReview-Commit-ID: 6vd8sb2zeuU
--HG--
extra : rebase_source : 25988c6790287e01fa7751effa72e8b924858948
llvm-symbolizer is necessary to get symbols in llvm-dsymutil crash
dumps. While we could use the one from clang during the build, it's
better if the llvm-dsymutil toolchain is standalone for local testing.
--HG--
extra : rebase_source : 5cd234a3e14ab52a4ce759821e0e756e68167797
When I originally wrote the llvm-dsymutil build script in bug 1430315,
I wasn't setting CMAKE_BUILD_TYPE to Release, and was ending up with
a very large binary (> 300MB), so I stripped it.
When I later set CMAKE_BUILD_TYPE to Release, I left the manual
stripping on, but that removes symbols that are useful for stacktraces
when dsymutil crashes (the Release type still leaves out debug info).
--HG--
extra : rebase_source : 802daadc24c0090574b1a44ea8b4e6c25735f703
By default, wget prints dots every 1k bytes. This can render a
lot of output for large files. We switch to the "mega" style, which
makes each dot represent 64k, thus reducing output by up to 64x.
We also force the use of dot display. By default, it uses "bar"
which attempts to use terminal formatting if possible. Since most
of this code executes in CI and terminal control characters can
interfere with logged output, we force the use of "dot." (Although
wget appears to automatically switch to dot in TC today. But
consistency is good.)
MozReview-Commit-ID: IpTWJdcauTV
--HG--
extra : rebase_source : 5c9aa1bbdcd78eaa0b31347ad026a2c1beaedc03
Now that we have a version of xz-utils that supports parallel compression,
we're not going to use it.
--HG--
extra : rebase_source : 7ac7840e979176b8d2042a4ef35f5868acd6f224
There were a few constraints in the choice of the version of dpkg to
backport:
- 1.17.20 is the first version that supports the debian source format
for that xz-utils package.
- versions >= 1.17.10 and <= 1.17.22 fail to build on wheezy.
- versions >= 1.17.21 depend on a version of patch not available on
wheezy.
All in all, the simpler choice was to go with version 1.17.20 with a
backport of the build failure fix.
That version of dpkg breaks the version of devscripts in wheezy, so the
version from wheezy-backports would be better to use, but we can't
unconditionally use it on all builds, because it happens that
mk-build-deps from that version is broken with the dpkg in wheezy.
In the end, it's simpler to build that backport and rely on package task
dependencies rather than selectively install the package from
wheezy-backports, so we do that. Except we can't use version
2.14.11~bpo70+1 because of bug 1419577.
--HG--
extra : rebase_source : 19ad1a44b770229fbc7e15bbcf01d3cb101315a8
In some cases, apt-get can't figure out how to install build
dependencies, and aptitude, with hints, is necessary instead. But
aptitude also doesn't necessarily work in some cases (e.g. it picks the
wrong automake version to build valgrind, although that's a problem in
valgrind's build dependencies). All in all, it's still better to use
apt-get for all the simple cases.
--HG--
extra : rebase_source : 5dcde3507f2642be8f284b6148d1ccaa4761164a
It can be necessary to use the artifacts from package tasks as build
dependencies for other package tasks.
--HG--
extra : rebase_source : a625670ae7af9a99c848924bdba9368c556e6766
Not all changes to debian_packages.py lead to actual changes to the
corresponding tasks. And since the tasks are pretty much self-contained,
we can consider that variations of them will be entirely represented in
the command used for the task. The only exception is the patch content
when there is an associated patch.
--HG--
extra : rebase_source : e70fa41a824106b5ceca09fd535c0c36bb0739ac
In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : 8c76bc91e1d5102f68c43e1050d61971fef32e9f
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 978cf033732cbbbb277d206dec69660175b82afa
Back in bug 1360609, we added `run-on-projects` to a list so that the
toolchain tasks wouldn't run on every push on release branches.
Fast forward to now, and they're depended upon by other tasks, meaning
they are triggered when appropriate, without resorting to that trick. In
fact, the commit message for bug 1360609 said we could switch to an
empty list once the jobs have dependencies.
The same is true from package tasks, which, in fact, I suspect would
happen on every push on release branches.
The only exception is for a few toolchains that are depended upon by
nothing, and that are produced for developer consumption with e.g. mach
artifact toolchain.
--HG--
extra : rebase_source : bb8624fed7490b85f4bd72b7ceb2db7a72b4c2ab
There are now only a handful of buildbot jobs remaining and the concern over
outdated treeherder exclusion profiles has largely been resolved.
This does remove the tc() group from a substantial number of tasks which will
now show up as top level tasks, potentially adding clutter. In some cases, we
might want to re-add a new group (e.g group builds or compiled tests together).
However rather than try to predict the best group names for tasks I'm unfamiliar
with, I think it's best to land this as is. Then if things are looking too
cluttered at the root namespace, file follow-up bugs as needed.
MozReview-Commit-ID: 8SMwjDwAOzV
--HG--
extra : rebase_source : 2f6d89d11c139bdcd404e7537db799d0e36ee4c3
When trying to remove an ubiquitous group like tc(), it's hard to tell where the
error was located without grepping my filesystem. This makes it a bit easier to
find and fix these errors.
MozReview-Commit-ID: 8NjvB5zOoqb
--HG--
extra : rebase_source : 167d3097f96548cf9c13b602d7d485cb69d48c2d
Relying on the various transforms setting it manually is error prone,
and, in fact, is why bug 1430037 busted beta. This change makes this
setting happen at a single place. This yields the same full task graph
as before, except for *more* chain-of-trust inputs being set now: they
were missing for toolchain tasks (which makes us closer to bug 1384430).
--HG--
extra : rebase_source : b6bf3a3b6da7174957c4c6814b853a51ee8a1e27
Add a geckoview-docs job that executes "./mach android geckoview-docs",
which takes care of calling gradle to generate the javadoc archive, and
uploading it to Github using given parameters.
MozReview-Commit-ID: DTWh4XdFZEO
--HG--
extra : rebase_source : 9d75be24cb553b3a773d3d34a2bdbdf4d4c8cd34
With the use of job-defaults, we can avoid a lot of repetition from
those definitions.
--HG--
extra : rebase_source : 932c2ed530aa8aec9a33da60cf652535fa0bd303
The mozharness transform is supposed to set the docker image to
desktop-build when not already set, but was not doing it properly.
I guess this is why some jobs were setting the image themselves, despite
using the mozharness transform.
Consequently, don't manually set the image to desktop-build when it's
the default.
--HG--
extra : rebase_source : 024bd10960bedaee3416785348a5c12498c5286f
At the same time, restrict the installed packages to the script
requirements to build Firefox. Toolchains have their own image so we
don't need to install packages for them.
--HG--
extra : rebase_source : c0e7aa178b1ce2ceb01f9dfe6af37bbb54d4d708
The one available in Debian wheezy is 1.7.10.4, which is really old, and
on our centos images, we're using 2.8.0rc3, which, while old too, is
more modern. While we may want to go with a more recent version, I'd
rather avoid differing from what we currently use, so use the exact same
version.
--HG--
extra : rebase_source : dfdf75a635073c248faef8a67648b2a83e4a1d84
... except libdmg-hfsplus. RedHat decided to patch libbz2 to have a
different soname, so a binary built on Debian can't run on
RedHat/CentOS. Ironically, a binary built on RedHat/CentOS can run
on Debian. While we could use some tricks to make libdmg-hfsplus built
on Debian work, at this point, it's not worth the effort. We can live
with libdmg-hfsplus being built on CentOS until the builds that use it
switch to Debian, which is imminent.
... and except mingw32-nsis. Sourceforce renewed their certificate last
week and somehow the corresponding CA is not yet recognized by the
ca-certificates in Debian wheezy (an update is underway but see below)
... and except wine, because it requires more 32-bits packages than can
be installed on the toolchain-build docker image. But all things
considered, the mingw32 builds don't need to be using the same docker
images as the linux builds, and they could be, like the android builds,
be based on a more recent build image. So the corresponding toolchains
can be built on a more recent version of Debian too.
Consequently, we keep all the mingw32 related toolchains on the
desktop-build image for now.
Because Ubuntu 16.04 changed in a way that busts gl3 tests and we can't
update the desktop1604-test image anymore.
--HG--
extra : amend_source : bfa07f9f77990dd6915b8c92d218227436bc6fc4
The install-mercurial.sh script currently installs a global mercurial
configuration after installing mercurial manually. In order to share
that configuration with docker images installing a mercurial package
through packages tasks, we move it to a separate file.
We however keep the part setting web.cacerts in install-mercurial.sh,
since it uses a path depending on what kind of environment the script is
run. Moreover, the instructions that come with mercurial to build
Debian packages come with web.cacerts set to the right path already, so
it's not needed in that case anyways.
At the same time, use multiple files in /etc/mercurial/hgrc.d/ instead
of a single /etc/mercurial/hgrc file.
--HG--
extra : rebase_source : 8140d8243cf012489025afe058f467c72224c891
Build dependencies won't be installed from backports unless they are not
satisfiable in the given Debian release. This is useful to get dh-python
on Wheezy.
--HG--
extra : rebase_source : 1f249b4ceae4fdd9ea37e9a9b9e9b62b48a1c9ed
In the case of mercurial, we don't want to use a .dsc as the original
source, but rather use the debian packaging scripts available in the
upstream mercurial tarball.
--HG--
extra : rebase_source : ec5b288f3994bc0bc1ec9ebce40def807bb7681f
The taskgraph.util.schema.check_schema function validates key names used
in schemas, ensuring they are dashed lower-case. However, it currently
assumes keys are either direct strings, Required or Optional entries,
and either ignores or fails to recognize other patterns.
For example, it ignores Any, and fails to recognize combinations like
Required(Any(...)), which we're going to use in next patch.
--HG--
extra : rebase_source : 4f6ff51a4a9dc9c7d9b6d070e03c6cc6e1befe80
I don't intend to install ccache in the Debian build images. Hazard
builds are the only builds running on the desktop-build image still
using ccache somehow, and that gains them nothing, since the ccache
directory is not set to a cached directory on taskcluster, meaning
the build always starts with an empty cache. If anything, this currently
makes the build slower.
Eventually, those builds should be able to use sccache, once the
necessary setup moves out of mozconfig.cache.
--HG--
extra : rebase_source : fba6dab78b25ea61892cbe6127ead36da395b0e0
taskcluster/taskgraph/transforms/task.py sets an expiry to 28 days for
tasks on try, vs. 1 year on other projects, but only do that when an
expiry is not already set, which docker images do. And they do always
set to 1 year.
But it doesn't make sense to keep large docker images from try for a
year. So use the same retention policy as the default one. We /could/
just remove the expiry from docker images and get the task.py default,
but it seems like whatever future change might happen to that default
shouldn't affect docker images, so it's better to duplicate the setting.
--HG--
extra : rebase_source : b5b46ca34a40ac82c5403b67d5b1aacf8cf8cceb
It was failing to build with the GCC/binutils on the CentOS-based docker
image, but it doesn't with the Debian-based one, so we can remove the
dependency on the gcc toolchain task. This allows sccache to remain
untouched when we change the gcc build scripts, and more importantly,
this allows it to depend on no toolchain that requires building things.
This makes it now possible to use sccache as a dependency for all other
toolchains jobs that compile, if that's beneficial (which might not be
the case, given the current sccache retention time, but at least it's a
viable option, now)
New Android-Gradle plugins pin the build-tools version, and we want to
be consistent between Gradle and moz.build.
MozReview-Commit-ID: ApWS4rHzPuH
--HG--
extra : rebase_source : 38a9781c472d858f3300cbbcbdc6d2311c465713
Turns out Google's maven repository doesn't publish checksums. I
can't imagine why not, but there it is. We have to think more about
whether to trust the artifacts downloaded from maven.google.com.
MozReview-Commit-ID: CdWijorq1IV
--HG--
extra : rebase_source : a884971e51ce7b1ff993754b130f462c476646ab
New Android-Gradle plugins pin the build-tools version, and we want to
be consistent between Gradle and moz.build.
MozReview-Commit-ID: ApWS4rHzPuH
--HG--
extra : rebase_source : 5a5730b4b9ce84af40a7c73c4f1abba017103f02
Turns out Google's maven repository doesn't publish checksums. I
can't imagine why not, but there it is. We have to think more about
whether to trust the artifacts downloaded from maven.google.com.
MozReview-Commit-ID: CdWijorq1IV
--HG--
extra : rebase_source : e4373273e7aea7df79d70b5fbc233233a84d2360
This marks **/docs/** as exclusively docs, and code that is autodoc'd as
inclusively docs.
That means that a change that purely modifies documentation files will *only*
run `docs` tasks, while a change that modifies autodoc'd source code will
*additionaly* run `docs` tasks. The tasks do not run by default.
MozReview-Commit-ID: G9tOK0AwtrI
--HG--
extra : rebase_source : 8dd971e5c9b0eb5f47895664a4ea49442f303ecb
extra : source : 0881de9b2b5e36ec37cc866f1d4af109da57a919
This makes use of pytest's generation feature. To add a new
template test, just add a new entry containing the input and
expected output to the dict in test_templates.py
MozReview-Commit-ID: 4qMefYHMjAp
--HG--
extra : rebase_source : ba3049885d1a2485048e1ff9913be43317559376
This is a minor cleanup of the python.yml source test tasks.
MozReview-Commit-ID: 6UanmbZHF8P
--HG--
extra : rebase_source : e06d310af9ca05bfdab1bc1e3bd2bc6aa3035cb9
This marks **/docs/** as exclusively docs, and code that is autodoc'd as
inclusively docs.
That means that a change that purely modifies documentation files will *only*
run `docs` tasks, while a change that modifies autodoc'd source code will
*additionaly* run `docs` tasks. The tasks do not run by default.
MozReview-Commit-ID: G9tOK0AwtrI
--HG--
extra : rebase_source : 8dd971e5c9b0eb5f47895664a4ea49442f303ecb
extra : source : 0881de9b2b5e36ec37cc866f1d4af109da57a919
And trigger a new Ubuntu 16.04 docker image with a different hash to get
the latest version of the GL drivers that were updated to llvmpipe 5
recently. Those new drivers make Firefox enabled
WEBGL_compressed_texture_s3tc, making the test pass.
--HG--
extra : rebase_source : 09a8995829f985aef29a8919fecaefaf90791584
We've had problems with crashes in llvm-dsymutil for a while, and while
they are, in essence, due to the fact that rustc produces bad debug
info, they are a hurdle to our builds. The tool comes along clang, and
updating clang is not necessarily easy (witness bug 1409265), so, so
far, we've relied on backporting fixes, which can be time confusing
(witness bug 1410148).
OTOH, llvm-dsymutil is a rather specific tool, that doesn't strictly
need to be tied to clang. It's only tied to it because it uses the llvm
code to do some of the things it does, and it's part of the llvm source
tree. But it could just as well be a separate tool, like it was(is?) on
OSX.
So, we add a toolchain job to build it from the llvm source,
independently from clang, so that we can update it separately, if we
hit new crashes that happen to already be fixed on llvm trunk. It will
also allow to more easily update after upstream fixes crashes after we
report them.
--HG--
extra : rebase_source : b814353b4b4632e46646a24b8f54c5300618ff49
In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : bbe8b001849e59bb655bb0e9766a6071ad38a52c
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 73e8fc51ea53af1e647fc1d5093c67d614dd009e
The one available in Debian wheezy is 3.81, but we're explicitly using
4.0 on CentOS, most notably because of its --output-sync option which
helps make logs better in some ways.
This takes the package from Debian jessie and builds it for Debian
wheezy.
--HG--
extra : rebase_source : 20bb550703fec41ed0175ef7f78c5b9a394160f3
This marks **/docs/** as exclusively docs, and code that is autodoc'd as
inclusively docs.
That means that a change that purely modifies documentation files will *only*
run `docs` tasks, while a change that modifies autodoc'd source code will
*additionaly* run `docs` tasks. The tasks do not run by default.
MozReview-Commit-ID: G9tOK0AwtrI
--HG--
extra : rebase_source : 8dd971e5c9b0eb5f47895664a4ea49442f303ecb
extra : source : 0881de9b2b5e36ec37cc866f1d4af109da57a919
This boils down to always setting worker.env to avoid a KeyError.
MozReview-Commit-ID: 1s4az9BFcc2
--HG--
extra : rebase_source : dabed2dedb00d176b829c6c0ff911e0236c5dec4
When looking for perfherder data collection duplicates, we currently
keep full job objects references, which are then used in case an error
occurs, to display the job names of the duplicates.
But those job objects are yielded and may be modified by other
transforms, and presently, by the time a duplicate is found, the
corresponding job object has been modified such that it has no 'name'
key anymore, leading to a KeyError exception when trying to display
the duplicate error message.
So instead of keeping the job objects, which can change, and which we
don't have a real use for, just keep the job name.
--HG--
extra : rebase_source : 204e90a6fe1e4ce62f361451e1176d3195a3383b
In bug 1427326, we added package tasks that can be depended upon by
docker image tasks. In that case, we add the routes containing a digest
for those package tasks when computing the docker image digests.
The problem is that those routes start with 'index.gecko.cache.level-n'
where n varies between try and e.g. mozilla-central. That means the
digest for those docker images varies between try and e.g.
mozilla-central, which then prevents try from using the cached versions
for mozilla-central when there is one, like for other docker images
without package dependencies.
What we really need from those routes is the digest part, which is
independent of the level, and we don't actually care about anything else
in the route string, so just use the digest.
--HG--
extra : rebase_source : 4aecf8472306963da34f2bd4d92675962c0432bc
This was useful when we still had buildbot-based build jobs, but all
it achieves nowadays is add friction when adding new build jobs on
taskcluster.
--HG--
extra : rebase_source : aa6a21a875eff1888c16900acf6d01ff37ab832b
New upstream release.
- Avoiding argument copies improves memory footprint.
- RwLock<T> no longer requires T to be Send.
- AsciiExt trait methods are now directly available
on str, [u8], u8, and char types without a `use`
statement.
MozReview-Commit-ID: 7Rx8uoNTMqH
--HG--
extra : rebase_source : 54068e34eaf6ccdbcc854fafb94d2a66fd068adf
There are e.g. some build infrastructure changes that we want to have a
controlled impact on the Firefox builds we produce. We have, in multiple
occasions, gone through manual work to compare Firefox builds, most of
the time using the diffoscope tool (https://diffoscope.org/).
This change introduces a new task kind that takes two Firefox builds as
input, either by name (reference to a build from the current task graph)
or by index (reference to a build from a previous push), and compares
them.
In order to get a Firefox build by index, we rely on dummy tasks with
an optimization we expect to always hit, so we add the necessary bits
to ensure those dummy tasks can go through up to the optimization phase
and be optimized out there.
--HG--
extra : rebase_source : 37482f67652dab2fcef2db4e6b8efe653999bae5
The spidermonkey mozjs and rust-bindings builds run sed on
$topsrcdir/.cargo/config.in to generate the cargo config they use, but
they previously only replaced the @top_srcdir@ substitution. This patch
makes them replace any other substitutions with an empty value to add
a bit of future-proofing.
MozReview-Commit-ID: 1DzP9vXxHMD
--HG--
extra : rebase_source : e8c0268a2a6e91ca2000b340beee2dcff0636591
We currently use a 32-bit Rust toolchain for win32 builds, but this can lead
to OOM situations. This patch makes win32 builds use a 64-bit Rust toolchain,
which requires a little bit of extra configuration because rustc needs to
be able to find a link.exe that produces 64-bit binaries for building
things like build scripts, which are host binaries.
We will now generate a batch file that sets LIB to the paths to 64-bit
libraries and invokes the x64-targeting link.exe, and add a section to the
.cargo/config file to instruct cargo to use that batch file as the linker
when producing 64-bit binaries.
MozReview-Commit-ID: 9vKBbm7Gvra
--HG--
extra : rebase_source : 599b3b661c7a8a5db1f32a2a9732fc202fb55e1e
Add a cross-compilation copy of rust's standard library targeting
i686-pc-windows-msvc to the win64-rust toolchain package so it
can be used to build for win32 as well.
MozReview-Commit-ID: 3598VZrDjIH
--HG--
extra : rebase_source : f1b25a68a67ae7f9c505a42d17f29dbedf59a49d
Instead of duplicating Dockerfiles between taskcluster/docker/*
directories, which can be error prone for very close images, it can be
desirable to use the same file. This change allows to set the
`definition` keyword on a docker image definition in kind.yml that
will make the task use the files from taskcluster/docker/<definition>
instead of taskcluster/docker/<image_name>.
--HG--
extra : rebase_source : 11ae231f66ca6a77896c1cff6c1580d04210f052
Ideally, we'd simply use the --build-arg docker argument along with ARG
in the Dockerfile, but that's only supported from Docker API 1.21, and
we're stuck on 1.18 for the moment.
So we add another hack to how we handle the Dockerfile, by adding a
commented syntax that allows to declare arguments to the Dockerfile.
The arguments can be defined in the docker images kind.yml file through
the `args` keyword. Under the hood, they are passed down to the docker
image task through the environment. The mach taskcluster-build-image
command then uses the corresponding values from the environment to
generate a "preprocessed" Dockerfile for its context.
--HG--
extra : rebase_source : 26a43dd680c1ab97b1a4689a23c55594a3b21b67
Landings of e.g. bug 1427336 triggered new toolchain jobs. One of those
jobs, because of wrong changes in bug 1421100, downloaded a new rust
compiler beta instead of the intended fixed beta version. In turn, that
new rust compiler beta fails to compile the slog crate.
Now, because of how toolchain cache indexes work, every new win32 job
picks that new unintended rust compiler beta version, even on branches
where 1427336 hasn't landed.
I couldn't find a way to force the right beta version, so we're pretty
much stuck with that toolchain index pointing to the wrong version of
rust beta.
By backing out bug 1426324, we return to a toolchain index that is known
to have an artifact for the right rust compiler beta.
Unfortunately, if something triggers a new TW32(rust) job after this,
that toolchain index will be busted as well.
Giving a directory to %include would copy all leaf files under one
single directory in the context image. The only image affected is
valgrind-build, which ended up having a dot-config/pip.conf file instead
of dot-config/pip/pip.conf, meaning valgrind jobs weren't using the
pip config.
--HG--
extra : rebase_source : 518c8ca1617b57ae4b4bb83a85340de5515c26c5
Android cppunit, test-verify, and mochitest-gpu have been running on lower end
aws instances. It is probably better to run all emulator tasks on xlarge.
libcrypto, part of OpenSSL, and that dmg links against, has a varying
ABI, and something built against libcrypto on Centos won't run on Debian
and vice versa. It might not even work between versions of the same OS
(e.g. Debian 7 vs. Debian 9).
Because of that, it is desirable to statically link it.
This incorporates https://github.com/mozilla/libdmg-hfsplus/pull/1
and sets OPENSSL_USE_STATIC_LIBS when building libdmg-hfsplus.
--HG--
extra : rebase_source : 21a46f707494f388a899c08d0923f8b393d12cd1
fontconfig uses expat by default to read its xml configuration, but when
expat is not there at build time, it falls back to libxml2. We ended up
in this situation, while on Debian, fontconfig is built against expat.
This makes no practical difference, since we're not actually using
fontconfig, but for some reason, the difference in dependencies has an
impact on how ld chooses to arrange the .plt and .got.plt sections,
meaning that even though the code and data is originally identical, in
the .o files, the resulting linked machine code is largely different
because of all the applied relocations changing the offsets of e.g. call
instructions for function calls through the .plt. This results in large
differences in .plt, .init, .text, .data.rel.ro, as well as symbols list
when building on Debian.
This thus is meant to help make the differences more tractable.
--HG--
extra : rebase_source : 7a731c34074a50e84412f73ab9499248345fb14a
OSX (cross) repackages are currently using a tooltool manifest to get
libdmg and hfsplus. Change those jobs to use the toolchain artifacts
instead.
At the same time, modify the repackage mozharness script's _run_tooltool
so that it doesn't fail with MOZ_TOOLCHAINS being set but without a
tooltool_manifest_src, matching the similar function in buildbase.py.
--HG--
extra : rebase_source : d128d4709c5d1d28d1a6b9c585fde82e99f725c7
I suppose it was setup through ~worker/.hgrc before we started
installing a /etc/mercurial/hgrc that enables a few other extensions
and sets some preferences.
There is no reason to now have two places where mercurial is being set
up, and it feels natural that we set it up at the system level.
Ideally, we'd also clean up the centos6-based images, but they require
an update of the centos6-build and centos6-build-upd images on the
docker hub, which is not really convenient, and those images are going
to be obsoleted soon anyways (bug 1399679).
--HG--
extra : rebase_source : 32c9cdf5d0fe8ac2c60a1c5a38e572c83a4783b2
Bug 1389715 removed the image definition in taskcluster/ci/docker-image
as well as the files associated with it in
taskcluster/docker/desktop-test, but the Dockerfile in there was the
only use of the Ubuntu 12.04 setup script, so it is currently unused.
--HG--
extra : rebase_source : 7d8018e7c94e2625ff9822a2d66231722a030394
The debian-build-system-setup.sh script doesn't use the install-make and
install-cmake scripts, so it's unnecessary to install them in /setup.
--HG--
extra : rebase_source : 4ba24b9827e67b9c7ad203e789e00e19d37786da
The script was added in bug 1179893 but looks like it has actually never
been used. A duplicate of the file was used for the upload-symbols
image, but that was removed in bug 1422740.
Since it was the only file in desktop-build/bin, we stop copying the
directory in docker images.
--HG--
extra : rebase_source : 4bcdb5ba0118e87455c6f596bf54e4528fe1b1ef
