Bug 1479533 was proposing to add a similar functionality, but this
iteration avoids actually unpacking anything, and ensures
reproducibility by relying on the reproducible bits from the original
archives: file ordering, flags, etc. (since they are checksummed, those
are never going to change for a given archive).
Another notable difference is that this applies the repack on the fetch
task itself, rather than create a separate task to apply the repack. The
latter has advantages, in that it allows to change the repacking without
redownloading the original file from a third-party server, but in
practice, most changes to the repacking would trigger the download tasks
anyways.
This patch only takes care of changing the archive type (zip->tar), and
the compression type (anything->zstandard).
Differential Revision: https://phabricator.services.mozilla.com/D40740
This also allows toolchain tasks to use aliases via fetches, which they
currently aren't allowed via use_toolchain. There are more toolchains
now than there were when the restriction was added, and it might be
useful to be able to use aliases. The flip side is that there are some
risks involved with aliases, which is why the restriction was there in
the first place. Let's see how things play out.
Differential Revision: https://phabricator.services.mozilla.com/D40713
What this means is that the sources for clang/llvm are downloaded
separately from the toolchain build (which also means we finally only
download a given version of clang once for all platforms).
In turn, this means the build-clang.py script needs to start with an
existing llvm-project tree, and we choose to make build-clang.py expect
that it's run from the llvm-project root directory.
This also means we don't need to download git for the windows toolchain
task.
Differential Revision: https://phabricator.services.mozilla.com/D40402
The build docker images need python-dev installed to build psutil, used
by the build resources monitor.
Differential Revision: https://phabricator.services.mozilla.com/D40781
--HG--
rename : python/mozbuild/mozbuild/resources/html-build-viewer/index.html => python/mozbuild/mozbuild/resources/html-build-viewer/build_resources.html
extra : moz-landing-system : lando
They are now all very similar, and there are only a few variations.
Differential Revision: https://phabricator.services.mozilla.com/D40365
MANUAL PUSH: avoid closing autoland while toolchains are rebuilt.
This removes the need for the hacks in wr-macos-cross-build-setup.sh and
wrench-windows-tests.sh, while keeping things working in other scripts.
Differential Revision: https://phabricator.services.mozilla.com/D40184
Bug 1563711 changed MOZ_FETCHES_DIR to make the MOZ_ANDROID_FAT_AAR_*
environment variables absolute paths. Unfortunately, the replacement
relies on non-osx/windows tasks to run in docker-worker, which is not
necessarily true, and that makes the MOZ_FETCHES_DIR wrong in
non-osx/windows generic-worker tasks. Apparently, that currently works,
but that's not guaranteed to stay this way.
The MOZ_ANDROID_FAT_AAR_* environment variables don't need to be
absolute paths, though. MOZ_FETCHES_DIR is normalized by run-task, and
MOZ_ANDROID_FAT_AAR_* can be set relative to that, which we do here.
Differential Revision: https://phabricator.services.mozilla.com/D40334
API lint is arguably the most valuable lint of all, but it's also hard
to fit into the Phab ecosystem, since there's no place to hang the
"API hash not correct" message in the case when the hash hasn't been
updated at all. Therefore, this commit doesn't convert it. See also
https://github.com/mozilla-mobile/gradle-apilint/issues/61 for adding
file/line information to API lint.
Differential Revision: https://phabricator.services.mozilla.com/D35277
--HG--
rename : mobile/android/config/mozconfigs/android-api-16-frontend/nightly => mobile/android/config/mozconfigs/android-api-16/nightly-android-lints
extra : moz-landing-system : lando
This was Gradle-only and then !Gradle-only. Now Gradle is required
and this is unused.
Differential Revision: https://phabricator.services.mozilla.com/D31381
--HG--
extra : moz-landing-system : lando
This bug fixes up some minor issues in the raptor taskcluster configuration by removing the comm-central repository, adding a comment about requiring --full when run-on-projects is [], removing the run-on-projects entry for raptor-tp6-3-firefox, and moving the android/opt run-on-projects definition closer to the android/pgo definition in the defaults section.
Differential Revision: https://phabricator.services.mozilla.com/D40282
--HG--
extra : moz-landing-system : lando
This was originally (I assume) used to control checking out build-tools, in a previous iteration of
our automation for checking out source. Our current tool doesn't look at this variable, and doesn't
support checking out build-tools.
Differential Revision: https://phabricator.services.mozilla.com/D40234
This turns on mochitests (except for gpu and remote) on Linux 64 (opt+debug) and
Win64 (opt only).
Differential Revision: https://phabricator.services.mozilla.com/D39917
--HG--
extra : moz-landing-system : lando
Plenty of places use `nproc`, and only a couple use `getconf
_NPROCESSORS_ONLN`. Use the former instead of the latter.
Differential Revision: https://phabricator.services.mozilla.com/D39999
--HG--
extra : moz-landing-system : lando
Add functionality for being able to send extra parameters for the test_url query of a test, directly from a taskcluster config.
Also, the PR adds logic in the `setup-raptor` taskgraph transform for dynamically changing the subset of youtube-playback tests based on the platform and project
Differential Revision: https://phabricator.services.mozilla.com/D39006
--HG--
extra : moz-landing-system : lando
Run fenix and refbrow raptor cold page-load tests once a day, through cron, at 3 AM.
Differential Revision: https://phabricator.services.mozilla.com/D39485
--HG--
extra : moz-landing-system : lando
This is loosely based on what was in bug 1467359, but simplified to
handle git only, and simply using git-archive because, at least now,
it's deterministic (it uses the commit date as timestamp in tar
archives).
This also adds 4 tasks for some of the things we use for toolchains, but
doesn't hook them up yet.
This also upgrades the fetch docker image to Debian buster, and installs
the required packages in it.
Differential Revision: https://phabricator.services.mozilla.com/D39480
Bug 1525373 had been waiting for a while for aarch64 windows fixes that
are still not there, and landed with a workaround for those. But while
waiting, bug 1555479 changed the run-task transform, making bug 1525373
double-wrap bitbar commands with /builds/taskcluster/script.py.
So remove the now redundant wrapping.
Differential Revision: https://phabricator.services.mozilla.com/D39821
--HG--
extra : moz-landing-system : lando
For some tasks, the workdir known to the decision task doesn't actually match
the workdir used in the task, which makes MOZ_FETCHES_DIR wrong when the
decision task derives it from the workdir.
On other tasks, MOZ_FETCHES_DIR is set to a relative directory, which
may work in some places where MOZ_FETCHES_DIR is used, but not in
others, that happen to be executed from a different directory.
To solve both problems, we set MOZ_FETCHES_DIR as a relative directory
everywhere, and we make run-task normalize it to an absolute path
before executing the task.
Differential Revision: https://phabricator.services.mozilla.com/D39152
--HG--
extra : moz-landing-system : lando
Avoid intermittently exceeding the already very long max-run-time in some chunks.
Differential Revision: https://phabricator.services.mozilla.com/D39679
--HG--
extra : moz-landing-system : lando
For some tasks, the workdir known to the decision task doesn't actually match
the workdir used in the task, which makes MOZ_FETCHES_DIR wrong when the
decision task derives it from the workdir.
On other tasks, MOZ_FETCHES_DIR is set to a relative directory, which
may work in some places where MOZ_FETCHES_DIR is used, but not in
others, that happen to be executed from a different directory.
To solve both problems, we set MOZ_FETCHES_DIR as a relative directory
everywhere, and we make run-task normalize it to an absolute path
before executing the task.
Differential Revision: https://phabricator.services.mozilla.com/D39152
--HG--
extra : moz-landing-system : lando
It took me several searchs after having looked here for how to create windows
interactive tasks. Add a link to the documentation here as a hint.
Differential Revision: https://phabricator.services.mozilla.com/D39568
--HG--
extra : moz-landing-system : lando
This adds the mobile project name, like "geckoview" or "fennec" to the Android
task labels. This makes the firefox/geckoview/fennec distinction more obvious
in 'mach try fuzzy' and 'mach try chooser'; it does not appear to affect
try syntax. This also adds "geckoview"/"fennec" to the job names displayed in
the lower-left treeherder pane. Hopefully this will help to clarify which
Android application is under test for each task.
Differential Revision: https://phabricator.services.mozilla.com/D39093
--HG--
extra : moz-landing-system : lando
Switch android-hw jsreftest to geckoview. Technically this runs all
android-hw reftests on geckoview -- I don't imagine we'll ever want to
run any android-hw fennec reftests.
Differential Revision: https://phabricator.services.mozilla.com/D39350
--HG--
extra : moz-landing-system : lando
In browsertime.zip we should have:
browsertime/
package.json
package-lock.json
node_modules/
.bin/
browsertime -> ../browsertime/bin/browsertime.js
browsertime/
...
The idea is that we'll fetch browsertime.zip in a generic-worker
environment and be able to run Node.js from within the top level
browsertime/ directory.
Differential Revision: https://phabricator.services.mozilla.com/D38773
--HG--
extra : moz-landing-system : lando
Increase test chunks to avoid intermittent task timeouts. At 18 chunks,
wpt-4 is the longest running, at about 90 minutes; 18 also aligns with
several /debug platforms.
Differential Revision: https://phabricator.services.mozilla.com/D39129
--HG--
extra : moz-landing-system : lando
This build re-uses the PGO profile from the win64 build in the
win64-aarch64-shippable-no-eme part of the aarch64 build. Even though
the profile isn't generated on the smae platform, we still get enough of
a performance win to make this worthwhile.
Note that the pgo_flags() in configure need to be tweaked slightly since
we don't supprt the -fprofile-generate flag for aarch64 (we don't build
the clang_rt.profile lib there). So we always want to return the flags
namespace to make sure we get the use_* versions of flags, which we do
need.
Differential Revision: https://phabricator.services.mozilla.com/D38928
--HG--
extra : moz-landing-system : lando
mozharness and mozharness test transforms for generic-worker currently
don't wrap the commands with run-task. This changes things such that the
commands are wrapped with run-task, by piggy-backing on the run_task
transform.
Some things then become redundant with what the run_task transform does,
and some others need to happen later than they currently do in order to
work.
Depends on D28026
Differential Revision: https://phabricator.services.mozilla.com/D28027
--HG--
extra : moz-landing-system : lando
This gets the wrench android builds working again after
07168db23c565e4506690612a7be50738844ddb2.
Differential Revision: https://phabricator.services.mozilla.com/D38882
--HG--
extra : moz-landing-system : lando
Our datadog contract is coming to an end. Soon we'll add influxdb data collection
Differential Revision: https://phabricator.services.mozilla.com/D38086
--HG--
extra : moz-landing-system : lando
We should test what we ship, and having talos benchmark the shipped
platform will help verify any PGO performance wins.
Differential Revision: https://phabricator.services.mozilla.com/D37896
--HG--
extra : moz-landing-system : lando
In order to run tests against the win64-aarch64-shippable build, we need
the test packages from the aarch64-no-eme build since the tests aren't
compiled during the -shippable stage. Since packages like cppunittest
and the target.test_packages.json file won't be generated correctly in
the -shippable stage, we disable the package-step tests there by setting
MOZ_AUTOMATION_PACKAGE_TESTS to 0 in the environment.
Differential Revision: https://phabricator.services.mozilla.com/D37895
--HG--
extra : moz-landing-system : lando
Not that it's causing any problem, but I noticed the build-base image
was being rebuilt when updating valgrind, and that seemed odd.
Differential Revision: https://phabricator.services.mozilla.com/D38297
--HG--
extra : moz-landing-system : lando
This fixes issues from bug 1565644 which caused speedometer to run on android-hw-p2-8-0-arm7-api-16/pgo on integration branches and android-hw-p2-8-0-arm7-api-16/pgo raptor tasks to run on mozilla-beta. This patch disables the aforementioned tests.
Differential Revision: https://phabricator.services.mozilla.com/D38111
--HG--
extra : moz-landing-system : lando
We've been lucky that non-sanitizer cross-builds for macosx have not
required the clang runtime so far, but they soon will. And it's only
available in the mac-cross clang toolchain, so we need to use that on
all macosx builds.
Differential Revision: https://phabricator.services.mozilla.com/D38260
--HG--
extra : moz-landing-system : lando
Changed the required yml, ini, json, js and html files to migrate ARES6 benchmark test to Raptor.
Differential Revision: https://phabricator.services.mozilla.com/D34178
--HG--
extra : moz-landing-system : lando
