This makes sure that the mozharness scripts have access to all the packages in
the build system virtualenv (namely mozbase).
Differential Revision: https://phabricator.services.mozilla.com/D22184
--HG--
extra : moz-landing-system : lando
This makes sure that the mozharness scripts have access to all the packages in
the build system virtualenv (namely mozbase).
Differential Revision: https://phabricator.services.mozilla.com/D22184
--HG--
extra : moz-landing-system : lando
We apply a local patch while we wait for upstream wine and mingw to review
the changes to widl that are necessary to generate the correct headers. Here we
just grab the generated headers and patch them into MinGW
We can revert this when MinGW updates, but for now we would like to unblock
the ANGLE update
Depends on D25294
Differential Revision: https://phabricator.services.mozilla.com/D25295
--HG--
extra : moz-landing-system : lando
This is needed to bring dispatcherqueue.h in, which is needed for
an ANGLE upgrade. It also ensures that overloads for secure string
functions are always defined and removes redundant --enable-secure-api
configure option and use of MINGW_HAS_SECURE_API
Differential Revision: https://phabricator.services.mozilla.com/D25294
--HG--
extra : moz-landing-system : lando
The updated cargo-apk version now correctly handles the `--frozen` flag
and additionally translates it to the `--offline` flag when invoking
gradle. This makes the gradle build fail instead of attempting network
fetches. To make the offline gradle build work, we set up a build.gradle
snippet that points to the maven repositories from the gradle toolchain
artifact, and have cargo-apk use that instead of the default jcenter()
repository.
Differential Revision: https://phabricator.services.mozilla.com/D24485
--HG--
extra : moz-landing-system : lando
No functional change here. This just extracts the existing script into a
helper file and shifts things around slightly so it's more logically
grouped (the env variables are needed for the cargo-apk invocation).
Also use better bash hygiene with variables.
Differential Revision: https://phabricator.services.mozilla.com/D24484
--HG--
extra : moz-landing-system : lando
The mozharness.py transform passes in "options" parameters through the
MOZHARNESS_OPTIONS environment variable. This will allow the Android PGO
run task to pass in the mozharness script name to test-linux.sh
Differential Revision: https://phabricator.services.mozilla.com/D22822
--HG--
extra : source : 097f141a499d151e167c85dcb57e66aade7c28cb
In order to call test-linux.sh with the job-script parameter, it needs
to have executable permissions.
Differential Revision: https://phabricator.services.mozilla.com/D22821
--HG--
extra : source : 6f5fc0d644dd1eb83294ce41b2b47be44c2d9783
The mozharness.py transform passes in "options" parameters through the
MOZHARNESS_OPTIONS environment variable. This will allow the Android PGO
run task to pass in the mozharness script name to test-linux.sh
Differential Revision: https://phabricator.services.mozilla.com/D22822
--HG--
extra : moz-landing-system : lando
In order to call test-linux.sh with the job-script parameter, it needs
to have executable permissions.
Differential Revision: https://phabricator.services.mozilla.com/D22821
--HG--
extra : moz-landing-system : lando
comm-task-env runs before run-task and updates the environment with GECKO_*
variables that are defined in a file at the root of the comm repository,
".gecko_rev.yml". run-task needs these variables to be set to find the
correct mozilla repository to check out for a particular TB build.
The current pinning method of updating ".taskcluster.yml" with the mozilla
repository and revision to pin tois no longer supported.
Differential Revision: https://phabricator.services.mozilla.com/D16783
--HG--
extra : moz-landing-system : lando
History does not disclose why we needed this, but in the brave new GCC
6-compiled world, deleting these files means that host links can no
longer find libstdc++, which causes problems. Let's put the files back.
Differential Revision: https://phabricator.services.mozilla.com/D22884
--HG--
extra : moz-landing-system : lando
We're going to copy an x86_64-unknown-linux-gnu ld into the clang build,
which clang will then use in preference to things on PATH. We therefore
need to ensure that this ld is the same ld as would be used for other
builds, such as PGO. This change is the most expedient way to do that;
future work will make the gcc job(s) depend on linux64-binutils directly.
Differential Revision: https://phabricator.services.mozilla.com/D22882
--HG--
extra : moz-landing-system : lando
Updating clang indicates that 32-bit compilation is substantially longer
than 64-bit compilation, perhaps due to swapping. The compilation
process is hitting the timeout limit shortly before the compilation
process completes (~3681/3695 tasks according to ninja).
We could tweak our clang build process to accommodate this job. But we
don't support building on 32-bit Windows anymore, and we don't produce a
32-bit Windows clang either. So we shouldn't support a 32-bit Windows
clang-tidy job either. Let's get rid of it.
Differential Revision: https://phabricator.services.mozilla.com/D23517
--HG--
extra : moz-landing-system : lando
History does not disclose why we needed this, but in the brave new GCC
6-compiled world, deleting these files means that host links can no
longer find libstdc++, which causes problems. Let's put the files back.
Depends on D22883
Differential Revision: https://phabricator.services.mozilla.com/D22884
--HG--
extra : moz-landing-system : lando
We're going to copy an x86_64-unknown-linux-gnu ld into the clang build,
which clang will then use in preference to things on PATH. We therefore
need to ensure that this ld is the same ld as would be used for other
builds, such as PGO. This change is the most expedient way to do that;
future work will make the gcc job(s) depend on linux64-binutils directly.
Depends on D22881
Differential Revision: https://phabricator.services.mozilla.com/D22882
--HG--
extra : moz-landing-system : lando
These toolchain tasks are the last ones using the historical
download-tools script from build/unix/build-gcc, which invokes gpg to
validate the downloaded tarballs. The consequence is that gpg-agent is
spawned and stays running, preventing a cleanup script from doing its
job, making the tasks fail.
Fetches are the new way to download sources, and can also do gpg
validation without those caveats.
The download-tools.sh script can then be removed as it's not used
anymore.
Differential Revision: https://phabricator.services.mozilla.com/D22682
--HG--
extra : moz-landing-system : lando
The build script currently is doing some unnecessary steps:
- Running `gclient` only prints out an help message. It is a step
indicated in various documentations, but is only necessary to keep
depot-tools up-to-date, which they are, since we just cloned it.
- The `fetch v8` command creates a v8 directory, no need to create
another layer.
- `gclient sync` is run as part of `fetch`. Same as `gclient`, this step
is only given in documentations to keep things up-to-date on an existing
clone, but we just freshly got one.
- Same goes for `git pull && gclient sync`
- `git checkout master` is not necessary, as `fetch` gets us there
already (albeit, in a detached head state)
- install-build-deps.sh installs build dependencies for chrome or
whatever. That's way too much for v8, that barely needs pkg-config and
glib, which we now install in the docker image.
Differential Revision: https://phabricator.services.mozilla.com/D20082
Make some adjustments to make the build work:
- On Linux, a newer GCC is needed, and -lrt is missing for
clock_gettime.
- On Mac, ninja expect AR be accept ar arguments, which clang doesn't,
so use llvm-ar.
- On Windows, remove the /WX flag that upstream sets (warnings as
errors) because there _are_ warnings in the code, and remove the
explicit /MACHINE:x64 linker flag because we're building for x86.
(we could build for x64, but I'd rather leave that to a followup)
Differential Revision: https://phabricator.services.mozilla.com/D19911
It turns out version 15.4.2 spawns a vctip process that sticks after the
build, and since bug 1527798, that breaks unmounting caches because the
process has a handle on msvcp140.dll, which lies in the source
directory. The problem goes away with 15.8.4, so upgrade all toolchain
tasks to that.
That's the same version as we're using on x86/x86-64 MSVC builds.
Differential Revision: https://phabricator.services.mozilla.com/D19752
to give time to docker images and toolchains to build.
--HG--
rename : taskcluster/docker/debian-raw/cloud-mirror-workaround.sh => taskcluster/docker/debian-base/cloud-mirror-workaround.sh
rename : taskcluster/docker/debian-raw/setup_packages.sh => taskcluster/docker/debian-base/setup_packages.sh
Make some adjustments to make the build work:
- On Linux, a newer GCC is needed, and -lrt is missing for
clock_gettime.
- On Mac, ninja expect AR be accept ar arguments, which clang doesn't,
so use llvm-ar.
- On Windows, remove the /WX flag that upstream sets (warnings as
errors) because there _are_ warnings in the code, and remove the
explicit /MACHINE:x64 linker flag because we're building for x86.
(we could build for x64, but I'd rather leave that to a followup)
Differential Revision: https://phabricator.services.mozilla.com/D19911
It turns out version 15.4.2 spawns a vctip process that sticks after the
build, and since bug 1527798, that breaks unmounting caches because the
process has a handle on msvcp140.dll, which lies in the source
directory. The problem goes away with 15.8.4, so upgrade all toolchain
tasks to that.
That's the same version as we're using on x86/x86-64 MSVC builds.
Differential Revision: https://phabricator.services.mozilla.com/D19752
Similar to the previous patch, this adds two jobs. One that
cross-compiles the binaries produced by `cargo test` and publishes them
as an artifact. The other job downloads the artifact and runs the
resulting binaries.
Differential Revision: https://phabricator.services.mozilla.com/D19370
--HG--
extra : source : 79120d13e713114032263c0851455ec5f981d72c
One job (which runs on a Linux docker worker) cross compiles wrench in
two configurations (regular and headless) and publishes artifacts with
the binaries. The other job (which runs on macOS) downloads the
artifacts and uses them to run the WebRender macOS CI release-mode
scripts, which basically consists of running the WebRender reftest
suite.
Differential Revision: https://phabricator.services.mozilla.com/D19369
--HG--
extra : source : 028c0ed368b844a4e54f3c38177b0fe65211d484
The shebang at the top of fetch-content doesn't work on macOS because
the path to python3 is not /usr/bin. Using /usr/bin/env doesn't work
properly on all platforms either so instead we invoke the script using
the currently running python executable.
Differential Revision: https://phabricator.services.mozilla.com/D19744
--HG--
extra : source : f6413e576a2169855f766085704570c9fc851ee2
Similar to the previous patch, this adds two jobs. One that
cross-compiles the binaries produced by `cargo test` and publishes them
as an artifact. The other job downloads the artifact and runs the
resulting binaries.
Differential Revision: https://phabricator.services.mozilla.com/D19370
--HG--
extra : moz-landing-system : lando
One job (which runs on a Linux docker worker) cross compiles wrench in
two configurations (regular and headless) and publishes artifacts with
the binaries. The other job (which runs on macOS) downloads the
artifacts and uses them to run the WebRender macOS CI release-mode
scripts, which basically consists of running the WebRender reftest
suite.
Differential Revision: https://phabricator.services.mozilla.com/D19369
--HG--
extra : moz-landing-system : lando
The shebang at the top of fetch-content doesn't work on macOS because
the path to python3 is not /usr/bin. Using /usr/bin/env doesn't work
properly on all platforms either so instead we invoke the script using
the currently running python executable.
Differential Revision: https://phabricator.services.mozilla.com/D19744
--HG--
extra : moz-landing-system : lando
libclang 3.9 has a bug that makes bindgen unable to distinguish some typedefs
from the underlying type, which matters for bug 1523071.
We have had quite a few workarounds for this bug and I don't really want to add
more, since in this case it is non-trivial. I think requiring libclang 4.0+ is
reasonable at this point.
Of the distros that can't build Firefox out of the box with clang, dropping support
for clang 3.9 would only break Ubuntu 14.04 LTS, which support ends April 2019,
right before we release 67.
Differential Revision: https://phabricator.services.mozilla.com/D18889
--HG--
rename : build/build-clang/clang-3.9-linux64.json => build/build-clang/clang-4.0-linux64.json
rename : taskcluster/scripts/misc/build-clang-3.9-linux.sh => taskcluster/scripts/misc/build-clang-4.0-linux.sh
extra : moz-landing-system : lando
This produces the same executables (produced for the same platforms) as
those currently pulled from tooltool (modulo timestamps, maybe changes
since last manifest change, etc.). Unfortunately, as of currently, the
Windows variant needs to be cross-built with mingw because it doesn't
compile without some POSIX APIs that MSVC/Windows SDK don't provide.
One thing that is left out of this change is whether to be completely
accurate with the toolchain cache hash (requiring a large list of files
as resources, and making those built very frequently), whether we'd
rely on manual updates, or if we should go with completely uncached
tasks. This can be left for a followup, the tasks not being hooked up
to be actually used by other tasks yet.
Differential Revision: https://phabricator.services.mozilla.com/D16302
--HG--
extra : moz-landing-system : lando
Two new kinds are introduced - 'instrumented-build' and
'generate-profile'. The instrumented-build kind is almost identical to
the build kind, except it can be used earlier in the task graph. The
3-tier PGO process becomes:
instrumented-build -> generate-profile -> build
The final build stage is identical to any other build, except it has
the 'use-pgo' flag set to True in its task definition. This flag causes
the transforms to add the instrumented-build and generate-profile tasks
to the taskgraph.
Differential Revision: https://phabricator.services.mozilla.com/D15750
--HG--
extra : moz-landing-system : lando
We create a minimal wrapper function to call collect_vcs_options()
and vcs_checkout().
We could consolidate this logic into vcs_checkout(). But I don't have
strong feelings about doing that.
Differential Revision: https://phabricator.services.mozilla.com/D13821
--HG--
extra : moz-landing-system : lando
One step closer to making all state gathering and normalization in one
place.
Differential Revision: https://phabricator.services.mozilla.com/D13819
--HG--
extra : moz-landing-system : lando
This makes behavior consistent across all VCS checkouts. I'm still not
a fan of using environment variables here. But at least this gets us
1 step closer to being able to plug alternate logic in without having
to update use of environment variables outside a single function.
Differential Revision: https://phabricator.services.mozilla.com/D13816
--HG--
extra : moz-landing-system : lando
We currently manage VCS checkout arguments as one-offs for each
project. This isn't scalable and results in a bit of copy-pasta.
Let's make the VCS checkout arguments generic so we can have the
same control for all repositories.
This commit focuses on consolidating the existing argument parser
code. It stops short of further unification, which will be done in
subsequent commits.
Differential Revision: https://phabricator.services.mozilla.com/D13815
--HG--
extra : moz-landing-system : lando
This code was used by mozharness jobs to check out the tools repo.
However, those jobs aren't actually using the repository.
Differential Revision: https://phabricator.services.mozilla.com/D13855
--HG--
extra : moz-landing-system : lando
We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.
Differential Revision: https://phabricator.services.mozilla.com/D13814
--HG--
extra : moz-landing-system : lando
We now have multiple things we may check out. "vcs" meaning "gecko"
is not obvious. Let's change the terminology to be more specific.
Differential Revision: https://phabricator.services.mozilla.com/D13813
--HG--
extra : moz-landing-system : lando
We create a minimal wrapper function to call collect_vcs_options()
and vcs_checkout().
We could consolidate this logic into vcs_checkout(). But I don't have
strong feelings about doing that.
Differential Revision: https://phabricator.services.mozilla.com/D13821
--HG--
extra : moz-landing-system : lando
One step closer to making all state gathering and normalization in one
place.
Differential Revision: https://phabricator.services.mozilla.com/D13819
--HG--
extra : moz-landing-system : lando
