Small updates to the version control documents regarding
upgrading `robustcheckout.py`. I noticed that we still
reference `build-puppet` from it's former home on hgmo
and don't include OpenCloudConfig in our list of places
we need to vendor `robustcheckout` changes.
Differential Revision: https://phabricator.services.mozilla.com/D23742
--HG--
extra : moz-landing-system : lando
We need to install a new enough binutils for both of these jobs and
ensure that it's properly found on the linux job.
Differential Revision: https://phabricator.services.mozilla.com/D23678
--HG--
extra : moz-landing-system : lando
History does not disclose why we needed this, but in the brave new GCC
6-compiled world, deleting these files means that host links can no
longer find libstdc++, which causes problems. Let's put the files back.
Differential Revision: https://phabricator.services.mozilla.com/D22884
--HG--
extra : moz-landing-system : lando
Firefox itself has moved on to GCC 6.x; we can move our toolchains along too.
Differential Revision: https://phabricator.services.mozilla.com/D22883
--HG--
extra : moz-landing-system : lando
We're going to copy an x86_64-unknown-linux-gnu ld into the clang build,
which clang will then use in preference to things on PATH. We therefore
need to ensure that this ld is the same ld as would be used for other
builds, such as PGO. This change is the most expedient way to do that;
future work will make the gcc job(s) depend on linux64-binutils directly.
Differential Revision: https://phabricator.services.mozilla.com/D22882
--HG--
extra : moz-landing-system : lando
(without the dash, because I want *this* push to build)
This filters out all tasks, but that means that several things will still run:
* docker images and tasks they depend on (debian packages)
* always_run tasks (various python-y things)
Differential Revision: https://phabricator.services.mozilla.com/D23020
--HG--
extra : moz-landing-system : lando
Updating clang indicates that 32-bit compilation is substantially longer
than 64-bit compilation, perhaps due to swapping. The compilation
process is hitting the timeout limit shortly before the compilation
process completes (~3681/3695 tasks according to ninja).
We could tweak our clang build process to accommodate this job. But we
don't support building on 32-bit Windows anymore, and we don't produce a
32-bit Windows clang either. So we shouldn't support a 32-bit Windows
clang-tidy job either. Let's get rid of it.
Differential Revision: https://phabricator.services.mozilla.com/D23517
--HG--
extra : moz-landing-system : lando
History does not disclose why we needed this, but in the brave new GCC
6-compiled world, deleting these files means that host links can no
longer find libstdc++, which causes problems. Let's put the files back.
Depends on D22883
Differential Revision: https://phabricator.services.mozilla.com/D22884
--HG--
extra : moz-landing-system : lando
Firefox itself has moved on to GCC 6.x; we can move our toolchains along too.
Depends on D22882
Differential Revision: https://phabricator.services.mozilla.com/D22883
--HG--
extra : moz-landing-system : lando
We're going to copy an x86_64-unknown-linux-gnu ld into the clang build,
which clang will then use in preference to things on PATH. We therefore
need to ensure that this ld is the same ld as would be used for other
builds, such as PGO. This change is the most expedient way to do that;
future work will make the gcc job(s) depend on linux64-binutils directly.
Depends on D22881
Differential Revision: https://phabricator.services.mozilla.com/D22882
--HG--
extra : moz-landing-system : lando
A newer clang may require newer binutils than the system provides, so we
should ensure that we provide just such a binutils.
Differential Revision: https://phabricator.services.mozilla.com/D23393
--HG--
extra : moz-landing-system : lando
Linux64/debug mochitest-dt chunks are increased to 16 to provide more capacity in
reasonable run time. For ccov, mochitest-dt-12 sometimes still runs > 90 minutes,
so ccov max-run-time is increased to 2 hours.
Differential Revision: https://phabricator.services.mozilla.com/D23309
--HG--
extra : moz-landing-system : lando
This will allow to use bash constructs in pre-diff-commands, like
`{a,b}`.
Depends on D23075
Differential Revision: https://phabricator.services.mozilla.com/D23076
--HG--
extra : moz-landing-system : lando
This is due to an incompatability somewhere between JaCoCo and
default interface methods.
Depends on D23016
Differential Revision: https://phabricator.services.mozilla.com/D23017
--HG--
extra : moz-landing-system : lando
This task extracts the binary of geckodriver from the
common test package and stores it into its own artifact.
For now this task is only run after Nightly build tasks
on supported platforms..
Differential Revision: https://phabricator.services.mozilla.com/D23094
--HG--
extra : moz-landing-system : lando
These toolchain tasks are the last ones using the historical
download-tools script from build/unix/build-gcc, which invokes gpg to
validate the downloaded tarballs. The consequence is that gpg-agent is
spawned and stays running, preventing a cleanup script from doing its
job, making the tasks fail.
Fetches are the new way to download sources, and can also do gpg
validation without those caveats.
The download-tools.sh script can then be removed as it's not used
anymore.
Differential Revision: https://phabricator.services.mozilla.com/D22682
--HG--
extra : moz-landing-system : lando
Reinsert awscli for partials, which is needed for caching. Also update packages and fix the metrics recording
Differential Revision: https://phabricator.services.mozilla.com/D22942
--HG--
extra : moz-landing-system : lando
Reinsert awscli for partials, which is needed for caching. Also update packages and fix the metrics recording
Differential Revision: https://phabricator.services.mozilla.com/D22942
--HG--
extra : moz-landing-system : lando
In bug 1501776, the `single_dep` and `multi_dep` schemas were aligned, which made both
branch in name_sanity identical. Simplify the code to reflect that.
Differential Revision: https://phabricator.services.mozilla.com/D24109
--HG--
extra : moz-landing-system : lando
This allows the confined snap to interact with Universal 2nd Factor devices, such as Yubikeys.
Differential Revision: https://phabricator.services.mozilla.com/D24147
--HG--
extra : moz-landing-system : lando
This introduces a mozharness script, android_emulator_pgo.py, to run the
profileserver suite with the PGO-instrumented Android build, and collect
the profile data and jarlog.
The mozharness script contains some redundancy with
build/pgo/profileserver.py, but the additional requirements for Android
to use adb and existing mozharness classes to control the emulator made
it difficult to share the desktop profileserver implementation.
Differential Revision: https://phabricator.services.mozilla.com/D22825
--HG--
extra : source : c151ebf303cad175e24bcc0965c800a9d12ecb3b
This is the first stage of the Android PGO task pipeline to generate an
instrumented build.
Differential Revision: https://phabricator.services.mozilla.com/D22824
--HG--
extra : source : b96dd954a456d8088a3ceda66f51d4106f516b4a
The mozharness.py transform passes in "options" parameters through the
MOZHARNESS_OPTIONS environment variable. This will allow the Android PGO
run task to pass in the mozharness script name to test-linux.sh
Differential Revision: https://phabricator.services.mozilla.com/D22822
--HG--
extra : source : 097f141a499d151e167c85dcb57e66aade7c28cb
In order to call test-linux.sh with the job-script parameter, it needs
to have executable permissions.
Differential Revision: https://phabricator.services.mozilla.com/D22821
--HG--
extra : source : 6f5fc0d644dd1eb83294ce41b2b47be44c2d9783
test-linux.sh defaults to true for NEED_XVFB, while build-linux.sh
defaults to false. If we are using test-linux.sh from mozharness (rather
than mozharness-test), we need to explicitly set NEED_XVFB to false in
order to not use xvfb.
Differential Revision: https://phabricator.services.mozilla.com/D22820
--HG--
extra : source : 53d3443e55d95af494d6c8bdc3d2d7a52c5eff1e
Most of our builds use libstdc++ compat, so they don't care much what
the custom toolchains we use are compiled with. The plain builds, on
the other hand, attempt to stick as closely as possible to a "local"
developer experience, and so don't set up libstdc++ compat. Since we
want to transition to our clang binaries being compiled with gcc 6, we
need a base system image that contains gcc 6 runtime libraries by
default. Debian 9 is just such a system.
Differential Revision: https://phabricator.services.mozilla.com/D22393
--HG--
extra : moz-landing-system : lando
Bug 1486071 intended to fix this, but while the tasks are setup to
restart on exit status 100, there are multiple failure cases where
snapshot.debian.org doesn't respond and the exit status is not 100.
One is dget, when downloading package sources from snapshot.debian.org.
Eventually, those should move to fetch tasks, but in the meantime, we
bubble up get errors with an exit code 100.
mk-build-deps wraps a call to apt-get install, but doesn't return the
exit code that apt-get returns when apt-get returns one. It makes it
hard to distinguish the error modes, but mk-build-deps is unlikely to
fail on anything else than apt-get. Not all apt-get failures would be
due to snapshot.debian.org availability, but that's a tradeoff we
decided was okay in bug 1486071.
Differential Revision: https://phabricator.services.mozilla.com/D22269
--HG--
extra : moz-landing-system : lando
Currently the scopes are handled in some test-specific code. However, there is
logic not to be in generic code.
Differential Revision: https://phabricator.services.mozilla.com/D22447
--HG--
extra : moz-landing-system : lando
This slightly decreases the amount of code that needs to know how to determine this.
Differential Revision: https://phabricator.services.mozilla.com/D22446
--HG--
extra : moz-landing-system : lando
Currently the scopes are handled in some test-specific code. However, there is
logic not to be in generic code.
Differential Revision: https://phabricator.services.mozilla.com/D22447
--HG--
extra : moz-landing-system : lando
This slightly decreases the amount of code that needs to know how to determine this.
Differential Revision: https://phabricator.services.mozilla.com/D22446
--HG--
extra : moz-landing-system : lando
As of the update snapshot, stretch-backports contains version 112.
Depends on D22264
Differential Revision: https://phabricator.services.mozilla.com/D22265
--HG--
extra : moz-landing-system : lando
This has the side effect of addressing bug 1524723 for those images.
Depends on D22263
Differential Revision: https://phabricator.services.mozilla.com/D22264
--HG--
extra : moz-landing-system : lando
When the apt snapshot is more recent than the docker image on the docker
hub, some packages may not be up-to-date.
Depends on D22455
Differential Revision: https://phabricator.services.mozilla.com/D22263
--HG--
extra : moz-landing-system : lando
Because the debian9-base apt configuration doesn't install recommended
packages, we end up needing to install more packages than before. We
could pass --install-recommended to apt-get, but that would make the
image larger than it already was after the upcoming changes, because
new versions of diffoscope come with more recommended dependencies.
The side effect is that this makes the image much smaller than it used
to be, while preserving all the useful recommended packages (we don't
actually need all of them).
Differential Revision: https://phabricator.services.mozilla.com/D22262
--HG--
extra : moz-landing-system : lando
I'd like to use the same format for config values, that get evaluated in different contexts, so don't
to mutate the config for that.
Differential Revision: https://phabricator.services.mozilla.com/D22126
--HG--
extra : moz-landing-system : lando
Artifact mozconfigs are not necessarily up-to-date wrt changes to the
nightly mozconfigs, and all in all, shouldn't be much different from
them.
It's just better to use the nightly mozconfigs (or beta on beta, etc.)
and make the mozconfigs themselves handle the few things that need to be
different when the USE_ARTIFACT environment is set (which is now
consistently set by taskcluster)
This does have the side effect of turning builds that actually don't
support artifact builds red when using --artifact on try, instead of
having them silently not be artifact builds as currently happens.
Depends on D21314
Differential Revision: https://phabricator.services.mozilla.com/D21315
--HG--
extra : moz-landing-system : lando
The artifact builds that are automatically derived using the artifact
template set the USE_ARTIFACT environment variable from taskcluster.
After the previous change, --artifact builds from try syntax do that
too.
That leaves us with only the artifact-build build not doing it, so for
consistency, do it there. That makes it not necessary to set
USE_ARTIFACT from mozconfig.artifact.automation anymore.
Depends on D22056
Differential Revision: https://phabricator.services.mozilla.com/D21313
--HG--
extra : moz-landing-system : lando
Currently, all tasks of kind builds are indiscriminately altered to use
artifacts, but only few of them actually support that, and the others
won't actually have the expected result when that happens. E.g. ASAN
builds with artifacts enabled end up being non-ASAN builds.
Effectively, this makes the artifact flag ignored for builds that don't
support artifacts. One could argue that those builds shouldn't happen at
all, but it feels a better use time of developer's time to just do the
full build they asked for. E.g. if they asked for ASAN with artifacts,
they still get an ASAN build, rather than an error or silently having
the task not happen after the decision task. This also allows to mix
artifact and non-artifact builds.
Further changes down the road are also modifying the artifact builds
configuration, which would actively turn those builds that don't support
artifact builds red (e.g. ASAN), so something has to be done anyways.
The alternative would be filter those builds out.
Depends on D21312
Differential Revision: https://phabricator.services.mozilla.com/D22056
--HG--
extra : moz-landing-system : lando
While try syntax is approaching its EOL, the fact that using it to do
artifact builds does some things subtly differently from using try
config is not helpful.
Depends on D22055
Differential Revision: https://phabricator.services.mozilla.com/D21312
--HG--
extra : moz-landing-system : lando
While the morph was changing the treeherder symbol to `Ba` for all jobs,
doing so with a transform fails because of the conflicting symbol check
(as multiple jobs in the same category would end up with `Ba`). So
instead, we append `a` to the existing symbol.
We also change the documentation wrt templates for try pushes, as the
artifact template is now essentially gone (although technically, mach
try will still set params['templates']['artifacts']['enabled'] for now,
and the template still exists, albeit empty).
Differential Revision: https://phabricator.services.mozilla.com/D22054
--HG--
extra : moz-landing-system : lando
