Adds command line option for developers to run tests against experimental debian 10 image (from D42597).
This is an experimental flag and will be removed once debian 10 image is used for production CI tests.
Differential Revision: https://phabricator.services.mozilla.com/D44236
--HG--
extra : moz-landing-system : lando
Stop running all Fennec functional (non-performance) tests:
- stop running all Android 4.3 tests
- switch android-em-7 cppunit and android-hw jittest from the Fennec apk to the
geckoview apk (no difference in behavior expected)
- stop running Android 7.0 marionette tests, since they also run against Fennec
- remove android-em-4.* references from taskcluster configs
- remove android instance: extra-large references from taskcluster configs,
since they only affect aws, which is no longer used for Android
Android-hw raptor tests running against Fennec remain; I will prepare a separate
patch for those.
Differential Revision: https://phabricator.services.mozilla.com/D43684
--HG--
extra : moz-landing-system : lando
The `./mach try {fuzzy,chooser}` commands now support a `--visual-metrics-jobs`
option which can be used to pass the job descriptions to the visual-metrics
task.
Differential Revision: https://phabricator.services.mozilla.com/D41878
--HG--
extra : moz-landing-system : lando
The `./mach try {fuzzy,chooser}` commands now support a `--visual-metrics-jobs`
option which can be used to pass the job descriptions to the visual-metrics
task.
Differential Revision: https://phabricator.services.mozilla.com/D41878
--HG--
extra : moz-landing-system : lando
For now, this just:
- updates the TreeHerder symbol
- includes the `--browsertime` extra option
The actual test jobs will react to the `--browsertime` extra option
and add dependencies, etc, as appropriate.
Differential Revision: https://phabricator.services.mozilla.com/D41605
--HG--
extra : moz-landing-system : lando
The current behaviour of resolve_tests_by suite is to look for exact
matches between the passed in path and the paths associated with each
test. But that doesn't work for cases where you want to path in an
ancestor path and run all the tests under that path.
This changeset makes resolve_tests_by_suite return a list of the input
paths that prefix-match one or more tests in a given suite, sorted by
suite. It is assumed that the harness itself will be able to handle
such inputs even when they don't correspond to a directory that
actualy contains tests.
Differential Revision: https://phabricator.services.mozilla.com/D42476
--HG--
extra : moz-landing-system : lando
Often times, PGO builds aren't required for testing things (in particular,
testing release automation). However, at least when testing release automation,
we do need to use the shippable build type.
Add an option to mach try that will disable using the 3-tier PGO jobs.
Differential Revision: https://phabricator.services.mozilla.com/D36365
--HG--
extra : moz-landing-system : lando
Templates invoke the `morph` logic, which is somewhat confusing and inflexible.
Update the machinery to support setting other `try_task_config` values.
Differential Revision: https://phabricator.services.mozilla.com/D36364
--HG--
extra : moz-landing-system : lando
Factor out the logic for calculating `try_task_config` from `push_to_try`,
so it can be called only for those selectors that need it.
Differential Revision: https://phabricator.services.mozilla.com/D36363
--HG--
extra : moz-landing-system : lando
Changes:
- require `--full` keyword for `./mach try fuzzy` in order to schedule android-hw jobs to hopefully reduce the backlog
Differential Revision: https://phabricator.services.mozilla.com/D33834
--HG--
extra : moz-landing-system : lando
- sm-shell: Selects shell only test cases that shouldn't require a full browser build.
- sm-all: Selects test cases that may require a full browser build.
Differential Revision: https://phabricator.services.mozilla.com/D28994
--HG--
extra : moz-landing-system : lando
Bug 1484243 removed the --git argument and made the VCS auto-detected,
so we should no longer suggest passing in that argument in error
messages.
Differential Revision: https://phabricator.services.mozilla.com/D30257
--HG--
extra : moz-landing-system : lando
Since we need to generate the full_task_set as a prereq to the target_task_set,
and since getting from full_task_set -> target_task_set is trivial.. we might
as well save both computed sets to the cache while we have them. This means users
that run:
$ ./mach try fuzzy
and then run:
$ ./mach try fuzzy --full
(or vice versa)
Will only incur task generation once. It also means that the 'watchman' trigger
will cache both taskgraphs.
Differential Revision: https://phabricator.services.mozilla.com/D29557
--HG--
extra : moz-landing-system : lando
This adds a 'watchman.json' file to /tools/tryselect and some documentation on
how to use it. Tl;dr, install watchman and then:
$ cd path/to/gecko
$ watchman -j < tools/tryselect/watchman.json
Differential Revision: https://phabricator.services.mozilla.com/D28771
--HG--
extra : moz-landing-system : lando
It turns out there are several places where the change to suite 'jittest-chunked'
causes problem. I am abandoning that approach.
Desktop uses this trick, and this returns android '-chunked' handling to a state
similar to what it was before I started messing around!
Differential Revision: https://phabricator.services.mozilla.com/D28897
--HG--
extra : moz-landing-system : lando
This skips a number of talos jobs that are unlikely to be affected
by due a browser-chrome specific change
Run with: `./mach try fuzzy --preset perf-chrome`
Differential Revision: https://phabricator.services.mozilla.com/D27911
--HG--
extra : moz-landing-system : lando
This officially makes 'moztest.resolve' the source of truth when it comes to
suite names. It aligns that file with the names used in both the
desktop_unittest and android_emulator_unittest scripts.
Differential Revision: https://phabricator.services.mozilla.com/D27555
--HG--
extra : moz-landing-system : lando
Currently we have the concept of a "suite" and a "flavour" in our task
configuration. Typically, the "suite" refers to the high-level test harness
like "mochitest" or "reftest", whereas the flavour is more specific, e.g
"browser-chrome-instrumentation" or "crashtest". However the line between suite
and flavour is not applied with any semblance of consistency which results in
inconsistent naming throughout the tree.
This patch gets rid of the concept of "flavours" entirely (at least when it
comes to task configuration). A suite is a type of test run, for example:
- mochitest-plain
- mochitest-devtools-chrome
- mochitest-browser-chrome-instrumentation
- jsreftest
- reftest
- firefox-ui-functional-remote
etc
There is no confusion here between suites and flavours because flavours don't
exist. However, there are a couple of places where we *do* need to know what
"test harness" is used to run a suite. These cases are:
1. For SCHEDULES moz.build rules
2. For the desktop_unittest.py mozharness script which takes arguments like
--mochitest-suite=browser (this is not a compelling use of this information
and should be refactored to work more like the android_emulator_unittest.py
script)
So to get this information, this patch introduces a new concept of a "category"
which is the overall "test harness" that runs the suite. For many suites, the
"category" is identical to the suite name. Unlike flavours, "categories" have
no bearing on how we call or refer to the suite.
Differential Revision: https://phabricator.services.mozilla.com/D27554
--HG--
extra : moz-landing-system : lando
This was regressed by bug 1544816 but the test never ran on the push that regressed.
This patch also updates the 'files-changed' for the tryselect task.
Differential Revision: https://phabricator.services.mozilla.com/D28386
--HG--
extra : moz-landing-system : lando
Changes:
- added windows10-aarch64 to the filter for fuzzy, to require `--full` in order to trigger jobs
- return False for any test tasks that contain windows10-aarch64 to prevent users using old try syntax from overwhelming the limited number of hardware
Differential Revision: https://phabricator.services.mozilla.com/D27590
--HG--
extra : moz-landing-system : lando
Since e10s is the default configuration, we shouldn't explicitly mark things
with the "-e10s" suffix. Instead we should mark things that *don't* run with
'e10s. This patch removes '-e10s' from all treeherder group symbols and task
labels, adds the "-1proc" suffix to tasks that are non-e10s.
Differential Revision: https://phabricator.services.mozilla.com/D25958
--HG--
extra : moz-landing-system : lando
This helps run the tests locally if fzf is normally installed in
$HOME/.mozbuild. Since the tests set MOZBUILD_STATE_PATH to a temporary
directory, fzf_bootstrap() can't find fzf in the HOME location unless it
is added to PATH.
Differential Revision: https://phabricator.services.mozilla.com/D27194
--HG--
extra : moz-landing-system : lando
