For simplicity, this implements just on in `NO_TASKS` (the default) or
on in `ALL_TASKS` (opt-in). This disables all category registrations
when in background task mode; we'll selectively re-enable things as
appropriate.
The flag constants were chosen to smoothly extend to a (16-)bit set in
the future, should we want to add a `JUST_TASKS("task", "other-task")`
option in the future.
This also adds ython tests for gen_static_components.py exercising
categories, simply 'cuz it's easiest to see what this adds in such
tests. Functional tests will follow in patches that actually
implement the new background tasks functionality.
Differential Revision: https://phabricator.services.mozilla.com/D96654
Per bug 1686507 comment 5, tier1 things should run on at least autoland and try. These tasks have too many dependencies that would need to start running on autoland, so I think it's not worth the trouble and I'd rather put them back to tier-2.
Differential Revision: https://phabricator.services.mozilla.com/D102336
This patch changes the formatting of the error lines output by the vismet tasks so they can be properly picked up by the Treeherder parser.
Differential Revision: https://phabricator.services.mozilla.com/D102763
remoteautomation.py is an old collection of code used by android mochitest and android reftest;
it survived the removal of automation.py. This patch removes remoteautomation.py, moving the
majority of the functionality to a new class in mozdevice. Some features are simplified or
removed, and the remainder moved into the remote mochitest/reftest harnesses.
Differential Revision: https://phabricator.services.mozilla.com/D102239
Now that we use an external dump_syms, we don't need to build
breakpad's.
This means we also don't need the dump_syms_rust_demangle crate anymore.
Differential Revision: https://phabricator.services.mozilla.com/D101865
The previous blockers for having this task are no longer an issue, as far as I can tell. The only perma-failure is bug 1612035. Until that's addressed we can remove those files with `pre-diff-commands` and have something rather than nothing.
Differential Revision: https://phabricator.services.mozilla.com/D101630
This patch makes it so that any retrigger actions being performed from Treeherder on browsertime tasks (through add-new-jobs, retrigger, and retrigger-multiple) will always retrigger the vismet (visual-metrics) tasks correctly. Furthermore, when a vismet task is retriggered, we'll retrigger the browsertime test task.
Differential Revision: https://phabricator.services.mozilla.com/D101404
rename_gcov_flush_clang_11.patch doesn't apply cleanly to clang 12 and will need to be forked. Turning the filename into a parameter will let us have the tasks exist side by side.
Differential Revision: https://phabricator.services.mozilla.com/D101495
Unlike the binary diffs, the generated-files diffs have been stable for a long time. If these tasks ever start failing, it would be good to learn about that at tier-1 speed.
Differential Revision: https://phabricator.services.mozilla.com/D101629
The BaseApp has been renamed to follow org.mozilla.firefox ID.
Additionally update the Docker image used to run the flatpak build.
Differential Revision: https://phabricator.services.mozilla.com/D100647
Passing all the directories to mach allows to take more advantage of
parallelism (plus not having to start up mach and the whole export once
for every directory, which itself takes a bit of time)
This takes:
GECKO_PATH=. taskcluster/scripts/misc/source-test-clang-non-unified.sh
from about ten mins to under two minutes on my machine.
Differential Revision: https://phabricator.services.mozilla.com/D101320
Now that linux64-clang-11-macosx-cross doesn't need a macosx64-clang-*
to build, there is no circular dependency to avoid.
Differential Revision: https://phabricator.services.mozilla.com/D101080
The new toolchain contains both aarch64 and x86_64 compiler-rts. We
could have a separate for each, but compiler-rt is small enough that
entirely separate toolchains is not warranted.
Differential Revision: https://phabricator.services.mozilla.com/D101079
The llvm build system does support building a universal compiler-rt for
multiple platforms at once, but as far as I know it only supports doing
so with the same SDK, while we want to use separate SDKs for each, so
build the x86_64 compiler-rt separately.
Differential Revision: https://phabricator.services.mozilla.com/D101078
We actually don't need the macosx-cross toolchain to build compiler-rt,
because building compiler-rt doesn't require compiler-rt (duh).
Differential Revision: https://phabricator.services.mozilla.com/D101077
The webrender wrench macos build, which is cross-compiled, needs a
macOS native libLLVM (a .dylib) to link against. The file is currently
part of the macosx-cross toolchain, but that was more incidental than
intentional. As we're going to change the macosx-cross toolchain in a
way that will remove the libLLVM.dylib, pull the file from the macOS
native clang.
Differential Revision: https://phabricator.services.mozilla.com/D101150
Only sanitizer builds require a native llvm-symbolizer executable.
Ideally, we'd build llvm-symbolizer from scratch, which would be faster,
but for now, let's go the easy route and just extract it from the
corresponding native clang builds.
We don't actually do anything with the llvm-symbolizer executable on
android builds, so we don't install it in $FINAL_TARGET, avoilding
the dependency on android builds (plus, we actually don't have an
android-native llvm-symbolizer, so even if it were already shipped, it
would be the wrong file).
Differential Revision: https://phabricator.services.mozilla.com/D101076
Now that linux64-clang-11-macosx-cross doesn't need a macosx64-clang-*
to build, there is no circular dependency to avoid.
Differential Revision: https://phabricator.services.mozilla.com/D101080
The new toolchain contains both aarch64 and x86_64 compiler-rts. We
could have a separate for each, but compiler-rt is small enough that
entirely separate toolchains is not warranted.
Differential Revision: https://phabricator.services.mozilla.com/D101079
The llvm build system does support building a universal compiler-rt for
multiple platforms at once, but as far as I know it only supports doing
so with the same SDK, while we want to use separate SDKs for each, so
build the x86_64 compiler-rt separately.
Differential Revision: https://phabricator.services.mozilla.com/D101078
We actually don't need the macosx-cross toolchain to build compiler-rt,
because building compiler-rt doesn't require compiler-rt (duh).
Differential Revision: https://phabricator.services.mozilla.com/D101077
Only sanitizer builds require a native llvm-symbolizer executable.
Ideally, we'd build llvm-symbolizer from scratch, which would be faster,
but for now, let's go the easy route and just extract it from the
corresponding native clang builds.
We don't actually do anything with the llvm-symbolizer executable on
android builds, so we don't install it in $FINAL_TARGET, avoilding
the dependency on android builds (plus, we actually don't have an
android-native llvm-symbolizer, so even if it were already shipped, it
would be the wrong file).
Differential Revision: https://phabricator.services.mozilla.com/D101076
In addition to the usual dot-release type of fixes, this also lets us drop a good amount of code that we had patched into our clang 11.
Differential Revision: https://phabricator.services.mozilla.com/D100959
Bug 1683824 updated to the 0.2.14 tag, but that's patched in order to
publish to crates.io such that tiny-http is not patched anymore.
Unfortunately, that breaks sccache-dist, which is a known issue and will
eventually fixed when addressing
https://github.com/mozilla/sccache/issues/912.
So here we update to a corresponding version on master (the first that
bumps the version to the next number).
Differential Revision: https://phabricator.services.mozilla.com/D100863
I was waiting for a better reason to do this, because the cbindgen
changes from 0.15.0 to 0.16.0 don't break trunk builds. But since
downstream has updated (see bug 1684180) and there's no reason not to,
let's do this to avoid future churn.
Differential Revision: https://phabricator.services.mozilla.com/D100499
Since e10s-multi is going to be enabled by default, this gives us the ability
to run junit in a secondary, e10s-single configuration so that we can see them
both side-by-side for a little while, at least until we're settled in with the
e10s-multi configuration.
Differential Revision: https://phabricator.services.mozilla.com/D94883
When building compiler-rt for macOS, its build system assumes the
compiler used is a native macOS clang, drops all the cmake C/C++ flags,
and adds its own.
Because we do need to pass `-target $target` (otherwise we end up with
ELF x86_64 objects) and `-mcpu=apple-a12` for the correct baseline for
arm64 macOS, and because there is unfortunately no cmake variable that
the compiler-rt build system will use, and because CMAKE_C*_COMPILER
need to be a program without arguments, we need to wrap the compiler.
While here, add `-v` to the ninja call to have more useful logs.
Differential Revision: https://phabricator.services.mozilla.com/D99841
This patch enables ASan for rust code in the `linux64-asan/{debug,opt}` and `linux64-asan-fuzzing` build tasks. Since it is a `-Z` unstable flag, we'll move to the same `linux64-rust-nightly` toolchain that TSan builds use.
For the time being, the ASan Nightly Reporter builds are left alone, until we're more familiar with the perf impacts of this landing.
Differential Revision: https://phabricator.services.mozilla.com/D99556
At this point it's pretty clear that we won't be reverting to clang-9.
This doesn't remove everything with clang-9 in the name. I did a mark-and-sweep GC by hand so this only removes unused entries. Some are still active, e.g. linux64-clang-9 is used to build a number of misc helper tools.
Differential Revision: https://phabricator.services.mozilla.com/D99721
This requires tests to specify `python-3: true` in order to be run
with Python 3. When nothing is specified things work just like today,
so it's a more conservative change than the one in bug 1672181.
Obviously in the long term we will remove this and switch to Python 3
only, but this unblocks moving harnesses to py3 today.
Differential Revision: https://phabricator.services.mozilla.com/D98766
Instantiating a wasm library duplicates a file descriptor for /dev/null 3 times to be used as input, output and error streams for the wasm sandboxed code. When a lot of sandboxes are created and destroyed, a lot of descriptors are duplicated and closed. While this should be fine, POSIX does not seem to happy with the opening and closing of many file descriptors --- this could perhaps be some strange interaction with Firefox's seccomp filters and cross-process file descriptor handling as it is difficult to repro this outside of firefox.
However, the simpler fix here was to just eliminate the duplication of /dev/null and return an error when input, output or error streams are accessed by wasm sandboxed code. This means calls to printf will fail, but no code I know off actually checks the int error code returned by printf and this change is certainly compatible with existing sandboxed components.
Differential Revision: https://phabricator.services.mozilla.com/D99160
This requires tests to specify `python-3: true` in order to be run
with Python 3. When nothing is specified things work just like today,
so it's a more conservative change than the one in bug 1672181.
Obviously in the long term we will remove this and switch to Python 3
only, but this unblocks moving harnesses to py3 today.
Differential Revision: https://phabricator.services.mozilla.com/D98766
This is step 1 to make the docs here better, as discussed. Another followup
will come in a day or two to update the text/links of the docs to refer to
this and the latest version of repack-rust.
Differential Revision: https://phabricator.services.mozilla.com/D99128
We ended up with a symbol `mda2-dafa26b89ed-bk-1754415059f-bk`, which is longer
than the allowed schema length of 25 characters. Let's keep the existing
backfill suffix per sheriffs.
Differential Revision: https://phabricator.services.mozilla.com/D98694
Triggered by the platform name change in bug 1673071 when macOS switched to
running its tests with WebRender enabled.
Differential Revision: https://phabricator.services.mozilla.com/D99095
clang 12 (specifically https://reviews.llvm.org/D91379) made some refactorings to libc++ that exposed a problem in the MinGW headers. That has now been fixed upstream.
In the meantime the headers also gained definitions for ProcessPayloadRestrictionPolicy, so we can remove our workaround for that.
Differential Revision: https://phabricator.services.mozilla.com/D98945
To build native python packages, the python dev packages are needed.
This should resolve the psutil installation failure.
Differential Revision: https://phabricator.services.mozilla.com/D98197
Removes a bunch of zstandard installations that were ran from
within the mach virtualenv, which should already have zstandard
installed.
Differential Revision: https://phabricator.services.mozilla.com/D98387
To ensure l10n updates are still picked up by beta builds in a timely
fashion, remove "DONTBUILD" from commit messages when running on
mozilla-beta, and run a couple of hours before the "daily-releases" job
starts.
Differential Revision: https://phabricator.services.mozilla.com/D98349
We keep dump-syms on 1.47 because it breaks at runtime when built with
1.48. It also puts it in sync with other toolchains that won't upgrade
until we need them to.
Differential Revision: https://phabricator.services.mozilla.com/D98422
The zstandard package is always installed in the mach virtualenv.
The patch assumes that zstandard is only used from the mach virtualenv,
and never the build virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D98387
While all toolkit and js-based projects make use of mfbt, some others,
like tools/crashreporter and tools/update-packaging, don't.
So instead of including mfbt from the top-level directory, include it
from the relevant project top-level mozbuilds.
This allows to remove the dependency on mfbt files in the hash for the
minidump-stackwalk and mar-tools toolchains.
Differential Revision: https://phabricator.services.mozilla.com/D98378
To ensure l10n updates are still picked up by beta builds in a timely
fashion, remove "DONTBUILD" from commit messages when running on
mozilla-beta, and run a couple of hours before the "daily-releases" job
starts.
Differential Revision: https://phabricator.services.mozilla.com/D98349
Bug 1672888 set up a plain opt build that runs on m-c only, like the
other plain opt builds. It didn't add plain debug builds on integration
branches as well to avoid having too many jobs running on the slim
mac worker pool, but since bug 1678154 moved most jobs off them, we can
now add those.
Differential Revision: https://phabricator.services.mozilla.com/D98279
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
To avoid breakage from Ubuntu package upgrades, we split the test image
into two, one that sets up the packages only, and that won't change when
we need to update our scripts, and another that derives from it, that
adds all our scripts and other setup.
Additionally, we work around the recent timeout issues due to the
upgrade of packages.
The timeout itself is due to gst-launch waiting indefinitely when it
crashes, rather than exiting with an error code. Bug 1679491 addresses
this issue, but the core problem is that gst-launch crashes, which seems
to be that some change in libc (presumably "Fix pthread_rwlock_try*lock
stalls") turns `gst_object_unref: assertion '((GObject *)
object)->ref_count > 0' failed` fatal warnings (which were already
happening) into actual crashes (presumably because a race condition is
lost on a use-after-free).
This workaround, however, will stop working as soon as the updated libc
package migrates from bionic-updates into bionic proper, presumaby on
the next 18.04 dot-release. Hopefully, we won't be rebuilding the base
image for a while, avoiding further problems. Eventually, we'll want to
upload the base image to docker hub so that it's set in stone, and
change the FROM in the base image to use that instead.
Differential Revision: https://phabricator.services.mozilla.com/D98045
