This patch fixes a failure with encodings when opening JSONs. It changes the file opening portion to ignore these types of errors - they are occuring because of the different language being used. We have no need for this data so we can safely ignore them. One other change that is being done here is adding additional error logging because it was possible to lose an error stack trace.
Differential Revision: https://phabricator.services.mozilla.com/D70939
This change is beneficial for two reasons:
1) Improvement on the single responsibility principle
2) Platform filter isn't 'bugbug' specific. E.g, the 'relevant tests' optimizer
could also theoretically use this. Having it as a standalone optimizer allows
us to compose it with other strategies.
Differential Revision: https://phabricator.services.mozilla.com/D71210
This ensures we don't run every build with every push via ./mach try auto. It
introduces a new 'optimization-overrides' try_config that can be used to
replace optimizations. For now, there is no user interface to pass this in via
the 'mach try' command line.
Differential Revision: https://phabricator.services.mozilla.com/D68207
Changes:
Applies the `filter_tasks_by_blacklist` method to try syntax pushes as well.
- moved `TARGET_TASK_BLACKLIST`and `filter_tasks_by_blacklist` method to live in `taskcluster/taskgraph/target_tasks.py`.
- removed existing filters against `ccov, windows10-aarch64` and `android-hw` filters against try syntax pushes.
- update imports for `fuzzy` and `chooser` selectors to refer to the new location of `filter_tasks_by_blacklist` method.
The reason for moving the logic (again) from `tools/tryselect` to `taskcluster/` is due to the placement of `try_option_syntax` and `target_tasks` files and both of those files handle the processing of `mach try syntax` pushes.
Differential Revision: https://phabricator.services.mozilla.com/D71698
The clang_rt.builtins lib files produced by the 1stage build do not include
functions necessary for 128bit integers. This functionality is required by the
OpenPGP libraries that Thunderbird is using, specifically Botan's bigint module.
When compiler-rt is built with MSVC, these functions are not included
by design.
Differential Revision: https://phabricator.services.mozilla.com/D71785
* For wrench builds, the clang toolchain must now be fetched and be
present in PATH for bindgen to work. When building OSMesa, we must
therefore set `LLVM_CONFIG=no` so that it does not attempt to build
llvmpipe.
* For wrench Mac cross-compiles, we must be careful to expose CFLAGS
and similar variables using the target-specific variable names, so
that host builds do not attempt to use flags intended for the
target. When building OSMesa we must use the generic variable `CC`,
so now we additionally set `HOST_CC`, so that host builds use the
host variable rather than the generic one.
* Similarily, for wrench android builds we must use a fork of
cargo-apk which sets the target-specific variables rather than the
generic ones. Otherwise we would attempt to use the NDK toolchain
for host builds.
Differential Revision: https://phabricator.services.mozilla.com/D70031
Firefox still does not default to Wayland when it's available (or when
X11 socket is not accessible). This can be changed by MOZ_ENABLE_WAYLAND=1
but as it's not the default, it's safer to just drop access to Wayland.
Differential Revision: https://phabricator.services.mozilla.com/D71448
Various updates to the custom retrigger action so that, without any custom changes to
parameters, the retriggered task runs with the same parameters as the original task.
Several issues were found and corrected, notably:
- parameters like --allow-software-gl-layers were ignored
- MOZHARNESS_TEST_PATHS was ignored
- many parameter customizations in the desktop mozharness configs were ignored
- mochitest suite/subsuite/flavor selection was not always correct
- using repeat=1 by default meant that each test ran twice
Differential Revision: https://phabricator.services.mozilla.com/D70457
Keeping the same for the currently chosen strategy for try auto, since we
don't want to decrease its regression detection rate.
We also add a new shadow scheduler which uses the reduced set with a higher
confidence threshold.
Differential Revision: https://phabricator.services.mozilla.com/D71205
This make four changes:
- use shippable/pgo builds on win64 and android platforms
- use linux64 instead of linux to get test packages for out-of-tree tasks
- consistently use `build` as the dependency name in tasks
- use the geckodriver toolchain, rather than the one packed in tasks
Differential Revision: https://phabricator.services.mozilla.com/D70943
Currently, we build android emulator packages manually and upload to
tooltool.
This patch switches it to be pulled from the toolchain built artifacts.
This also allows android tests to run in the staging environment.
Differential Revision: https://phabricator.services.mozilla.com/D68617
This patches fixes several problems found on Raptor and the condprof:
Raptor:
- Make sure the conditioned profile dir is removed after
it's been used, not before.
- Adds the --project option to raptor so we know if we're on try
autoland or mozilla-central.
- Both Fennec and Fenix are deactivated for now
- Use the allow-downgrade flag to be flexible on build ids (the next step will be bug 1628666)
Conditioned profiles, curation of the profile prefs:
- Fully deactivates Normandy during Raptor tests (app.normandy.enabled)
- Removes any GFX blacklisting (gfx.blacklist.*)
- Removes any marionette pref
- Enforce extensions sideloading (extensions.startupScanScopes)
Differential Revision: https://phabricator.services.mozilla.com/D70518
Changes:
- update the nodejs download path.
- save the file with a specific name and use that name going forward to reduce number of times the same filename is referenced.
Differential Revision: https://phabricator.services.mozilla.com/D71088
--HG--
extra : moz-landing-system : lando
We'll want some kind of backstop no matter what optimization algorithm we use.
We don't want to go too long without running any given task so we can find
regressions quickly and have a good merge candidate.
This pulls the logic that handles this out of the SETA strategy and into its
own strategy.
This will also make the SETA shadow scheduler more representative of what the
algorithm is doing.
Note in the future we may find ways to make this backstop more efficient (i.e
only run tasks that didn't run in the last 9 pushes for example).
Depends on D68621
Differential Revision: https://phabricator.services.mozilla.com/D68622
--HG--
extra : moz-landing-system : lando
This allows to nest strategies without having to register ever intermediate
composite strategy first. For example:
All(Any("skip-unless-schedules", "seta"), "backstop")
Prior to this patch, we'd need to register that 'Any' one first and then use it
in the 'All'.
Depends on D68620
Differential Revision: https://phabricator.services.mozilla.com/D68621
--HG--
extra : moz-landing-system : lando
I'd like to implement a 'backstop' strategy, such that it will prevent all other
optimizers from removing tasks under certain conditions (e.g every 10th push).
The nicest way to implement this seems to be an 'All' composite strategy
(similar to 'Either' which this patch renames to 'Any'). This means we could
do something like:
All("seta", "backstop")
which means we would only remove tasks if *all* substrategies say to remove
tasks.
Differential Revision: https://phabricator.services.mozilla.com/D68620
--HG--
extra : moz-landing-system : lando
We'll want some kind of backstop no matter what optimization algorithm we use.
We don't want to go too long without running any given task so we can find
regressions quickly and have a good merge candidate.
This pulls the logic that handles this out of the SETA strategy and into its
own strategy.
This will also make the SETA shadow scheduler more representative of what the
algorithm is doing.
Note in the future we may find ways to make this backstop more efficient (i.e
only run tasks that didn't run in the last 9 pushes for example).
Depends on D68621
Differential Revision: https://phabricator.services.mozilla.com/D68622
--HG--
extra : moz-landing-system : lando
This allows to nest strategies without having to register ever intermediate
composite strategy first. For example:
All(Any("skip-unless-schedules", "seta"), "backstop")
Prior to this patch, we'd need to register that 'Any' one first and then use it
in the 'All'.
Depends on D68620
Differential Revision: https://phabricator.services.mozilla.com/D68621
--HG--
extra : moz-landing-system : lando
I'd like to implement a 'backstop' strategy, such that it will prevent all other
optimizers from removing tasks under certain conditions (e.g every 10th push).
The nicest way to implement this seems to be an 'All' composite strategy
(similar to 'Either' which this patch renames to 'Any'). This means we could
do something like:
All("seta", "backstop")
which means we would only remove tasks if *all* substrategies say to remove
tasks.
Differential Revision: https://phabricator.services.mozilla.com/D68620
--HG--
extra : moz-landing-system : lando
These configs now have the same `fetches`/`toolchain`/`by-test-platform` values
as the `xpcshell` config in `xpcshell.yml`.
Differential Revision: https://phabricator.services.mozilla.com/D70650
--HG--
extra : moz-landing-system : lando
Currently the linux64-clang-9-win-cross toolchain depends on the win64-clang-cl toolchain for a few files. This causes very long lead times when toolchains are rebuilt, because of the un-parallelizable chain of tasks win64-clang-cl -> linux64-clang-9-win-cross -> builds.
As a partial mitigation, this patch adds a single-stage clang-cl build for consumption by the cross toolchain. It's not a very high quality build, but good enough for the purpose it serves, while being faster to build.
Differential Revision: https://phabricator.services.mozilla.com/D70254
--HG--
rename : build/build-clang/clang-win64.json => build/build-clang/clang-win64-1stage.json
extra : moz-landing-system : lando
The goal being to detect potential build failures for smoosh on a tier 1 build.
Differential Revision: https://phabricator.services.mozilla.com/D69621
--HG--
extra : moz-landing-system : lando
The goal being to detect potential build failures for smoosh on a tier 1 build.
Differential Revision: https://phabricator.services.mozilla.com/D69621
--HG--
extra : moz-landing-system : lando
Activates the conditioned profile by doing the following changes:
- make sure the conditioned profile dir is removed after
it's been used, not before.
- add the --project option to raptor so we know if we're on try
or mozilla-central.
- Both Fennec and Fenix are deactivated for now.
- Remove any gfx.blacklist.* prefs when using a conditioned profile
Differential Revision: https://phabricator.services.mozilla.com/D70518
--HG--
extra : moz-landing-system : lando
The `video_location` attribute (which was the URL or path of the video) was
removed from the `Job` type, due to it no longer being possible to be a URL.
This was because videos are now passed in through the browsertime-results.tgz
artifact, instead of specified separately in the job description.
This error message was missed in the refactor and was causing failures due to
visualmetrics.py to appear to be caused by our wrapper script. This will not
fix the underlying error causing the intermittents in the first place, but now
the real cause will appear in the log instead.
Differential Revision: https://phabricator.services.mozilla.com/D70062
--HG--
extra : moz-landing-system : lando
Make sure the conditioned profile dir is removed after
it's been used, not before. This patch also adds the
--project option to raptor so we know if we're on try
or mozilla-central. Both Fennec and Fenix are deactivated
for now.
Differential Revision: https://phabricator.services.mozilla.com/D70518
--HG--
extra : moz-landing-system : lando
Since we don't look at the changesets, there is no need for hgmo to generate
them for us.
Differential Revision: https://phabricator.services.mozilla.com/D70567
--HG--
extra : moz-landing-system : lando
CLOSED TREE
Backed out changeset f42150bdee2b (bug 1626772)
Backed out changeset ab5b637f714a (bug 1626772)
Backed out changeset fd4026a9f380 (bug 1626772)
For building `macosx64-clang-tidy` with `linux64-clang-10` we need to build `cctools`
with `clang-10`.
Differential Revision: https://phabricator.services.mozilla.com/D70064
--HG--
extra : moz-landing-system : lando
Patches that are applied on top on `clang-10` repo are based on the original
patches from our trunk and have been rebased on top of `clang-10`.
Please see as an example: `find_symbolizer_linux.patch` copied to `find_symbolizer_linux_clang_10.patch`.
Differential Revision: https://phabricator.services.mozilla.com/D70063
--HG--
rename : build/build-clang/clang-linux64.json => build/build-clang/clang-10-linux64.json
rename : build/build-clang/find_symbolizer_linux.patch => build/build-clang/find_symbolizer_linux_clang_10.patch
rename : build/build-clang/llvmorg-11-init-4265-g2dcbdba8540.patch => build/build-clang/llvmorg-11-init-4265-g2dcbdba8540_clang_10.patch
rename : build/build-clang/rG7e18aeba5062.patch => build/build-clang/rG7e18aeba5062_clang_10.patch
rename : build/build-clang/rename_gcov_flush.patch => build/build-clang/rename_gcov_flush_clang_10.patch
rename : build/build-clang/tsan-hang-be41a98ac222.patch => build/build-clang/tsan-hang-be41a98ac222_clang_10.patch
rename : build/build-clang/unpoison-thread-stacks.patch => build/build-clang/unpoison-thread-stacks_clang_10.patch
extra : moz-landing-system : lando
Previously source-tests requiring a build had a "global" mapping of
platform to build type in kind.yml. But this made it confusing to
figure out how to add task-specific configuration. To simplify things,
make the configuration for the dependent platforms also go in the task
definition.
Differential Revision: https://phabricator.services.mozilla.com/D68636
--HG--
extra : moz-landing-system : lando
The `linux64-clang-9-cross` toolchain was forked from `linux64-clang-9` in https://hg.mozilla.org/integration/autoland/rev/ca923afe3ed4 in order to pick up a patch for Windows compilation.
This is no longer necessary for two reasons:
1. Windows builds stopped depending on that patch in bug 1608460.
2. Even if we still needed that patch, the `linux64-clang-9` toolchain gained all of the Windows patches in bug 1618473.
Therefore the `clang-dist` task can go back to using unmodified `linux64-clang-9`.
Differential Revision: https://phabricator.services.mozilla.com/D70263
--HG--
extra : moz-landing-system : lando
For building `macosx64-clang-tidy` with `linux64-clang-10` we need to build `cctools`
with `clang-10`.
Differential Revision: https://phabricator.services.mozilla.com/D70064
--HG--
extra : moz-landing-system : lando
Patches that are applied on top on `clang-10` repo are based on the original
patches from our trunk and have been rebased on top of `clang-10`.
Please see as an example: `find_symbolizer_linux.patch` copied to `find_symbolizer_linux_clang_10.patch`.
Differential Revision: https://phabricator.services.mozilla.com/D70063
--HG--
rename : build/build-clang/clang-linux64.json => build/build-clang/clang-10-linux64.json
rename : build/build-clang/find_symbolizer_linux.patch => build/build-clang/find_symbolizer_linux_clang_10.patch
rename : build/build-clang/llvmorg-11-init-4265-g2dcbdba8540.patch => build/build-clang/llvmorg-11-init-4265-g2dcbdba8540_clang_10.patch
rename : build/build-clang/rG7e18aeba5062.patch => build/build-clang/rG7e18aeba5062_clang_10.patch
rename : build/build-clang/rename_gcov_flush.patch => build/build-clang/rename_gcov_flush_clang_10.patch
rename : build/build-clang/tsan-hang-be41a98ac222.patch => build/build-clang/tsan-hang-be41a98ac222_clang_10.patch
rename : build/build-clang/unpoison-thread-stacks.patch => build/build-clang/unpoison-thread-stacks_clang_10.patch
extra : moz-landing-system : lando
This patch changes the test that browsertime runs for speedometer from `raptor-speedometer-APP` to `speedometer` and changes some other small issues like the missing app name in the `extraOptions` field in the perfherder data.
Differential Revision: https://phabricator.services.mozilla.com/D69581
--HG--
extra : moz-landing-system : lando
Note that it only runs jit-tests (not jsapi-tests and the slower jstests) and
only two jit-test configurations. This makes the job very fast, I see it finish
in just 12-13 minutes on Try.
When this lands it should be green. For now it's tier 2 and only runs on central.
When WarpBuilder gets closer to shipping we can consider changing that (and probably
run more tests).
Differential Revision: https://phabricator.services.mozilla.com/D69512
--HG--
extra : moz-landing-system : lando
This patch disabled chrome release tests on mozilla-central and has them run through the cron task instead.
Differential Revision: https://phabricator.services.mozilla.com/D69570
--HG--
extra : moz-landing-system : lando
