Templates invoke the `morph` logic, which is somewhat confusing and inflexible.
Update the machinery to support setting other `try_task_config` values.
Differential Revision: https://phabricator.services.mozilla.com/D36364
--HG--
extra : moz-landing-system : lando
Factor out the logic for calculating `try_task_config` from `push_to_try`,
so it can be called only for those selectors that need it.
Differential Revision: https://phabricator.services.mozilla.com/D36363
--HG--
extra : moz-landing-system : lando
It turns out there are several places where the change to suite 'jittest-chunked'
causes problem. I am abandoning that approach.
Desktop uses this trick, and this returns android '-chunked' handling to a state
similar to what it was before I started messing around!
Differential Revision: https://phabricator.services.mozilla.com/D28897
--HG--
extra : moz-landing-system : lando
This officially makes 'moztest.resolve' the source of truth when it comes to
suite names. It aligns that file with the names used in both the
desktop_unittest and android_emulator_unittest scripts.
Differential Revision: https://phabricator.services.mozilla.com/D27555
--HG--
extra : moz-landing-system : lando
Currently we have the concept of a "suite" and a "flavour" in our task
configuration. Typically, the "suite" refers to the high-level test harness
like "mochitest" or "reftest", whereas the flavour is more specific, e.g
"browser-chrome-instrumentation" or "crashtest". However the line between suite
and flavour is not applied with any semblance of consistency which results in
inconsistent naming throughout the tree.
This patch gets rid of the concept of "flavours" entirely (at least when it
comes to task configuration). A suite is a type of test run, for example:
- mochitest-plain
- mochitest-devtools-chrome
- mochitest-browser-chrome-instrumentation
- jsreftest
- reftest
- firefox-ui-functional-remote
etc
There is no confusion here between suites and flavours because flavours don't
exist. However, there are a couple of places where we *do* need to know what
"test harness" is used to run a suite. These cases are:
1. For SCHEDULES moz.build rules
2. For the desktop_unittest.py mozharness script which takes arguments like
--mochitest-suite=browser (this is not a compelling use of this information
and should be refactored to work more like the android_emulator_unittest.py
script)
So to get this information, this patch introduces a new concept of a "category"
which is the overall "test harness" that runs the suite. For many suites, the
"category" is identical to the suite name. Unlike flavours, "categories" have
no bearing on how we call or refer to the suite.
Differential Revision: https://phabricator.services.mozilla.com/D27554
--HG--
extra : moz-landing-system : lando
This was regressed by bug 1544816 but the test never ran on the push that regressed.
This patch also updates the 'files-changed' for the tryselect task.
Differential Revision: https://phabricator.services.mozilla.com/D28386
--HG--
extra : moz-landing-system : lando
Since e10s is the default configuration, we shouldn't explicitly mark things
with the "-e10s" suffix. Instead we should mark things that *don't* run with
'e10s. This patch removes '-e10s' from all treeherder group symbols and task
labels, adds the "-1proc" suffix to tasks that are non-e10s.
Differential Revision: https://phabricator.services.mozilla.com/D25958
--HG--
extra : moz-landing-system : lando
This helps run the tests locally if fzf is normally installed in
$HOME/.mozbuild. Since the tests set MOZBUILD_STATE_PATH to a temporary
directory, fzf_bootstrap() can't find fzf in the HOME location unless it
is added to PATH.
Differential Revision: https://phabricator.services.mozilla.com/D27194
--HG--
extra : moz-landing-system : lando
Add Android 7.0 gtests, opt and debug, running against the geckoview
TestRunnerActivity.
Differential Revision: https://phabricator.services.mozilla.com/D27016
--HG--
extra : moz-landing-system : lando
There are a few windows/mac-only suites that were missed since we were only
reading the linux variant.
Depends on D25401
Differential Revision: https://phabricator.services.mozilla.com/D25432
--HG--
extra : moz-landing-system : lando
While running presets + intersection queries works with 'mach try fuzzy
--preset <foo>', it is still broken with 'mach try --preset <foo>'.
Differential Revision: https://phabricator.services.mozilla.com/D25298
--HG--
extra : moz-landing-system : lando
Some of these were working with the '<flavor>-<subsuite>' mechanism that was
previously being used, but better to be explicit wherever possible.
Depends on D25077
Differential Revision: https://phabricator.services.mozilla.com/D25078
--HG--
extra : moz-landing-system : lando
Mozharness appends -chunked/-coverage to some suites, but the build system/test
resolver don't have any concept of these things. We need to normalize these out
for the purposes of MOZHARNESS_TEST_PATHS.
Differential Revision: https://phabricator.services.mozilla.com/D25015
--HG--
extra : moz-landing-system : lando
It turns out bug 1489100 regressed the ability to specify test paths for most
suites by naively assuming that mozharness uses suite names that look like:
<flavor>-<subsuite>
In reality, there is no consistency to how suite names are generated. This test
does a few things:
1) Patches the moztest.TestResolver to return a list of all possible
suites/subsuites (assuming the lists in moztest.resolve are up to date).
2) Finds all the suites defined in the mozharness configs (e.g
linux_unittest.py), and uses these are parametrized inputs.
3) Checks that for each test suite,
DesktopUnittest._get_mozharness_test_paths() returns something.
I've marked all of the test suites that currently fail as expected. This way I
have a good sense of what needs to be fixed, and can validate my changes as I
move through the list.
Differential Revision: https://phabricator.services.mozilla.com/D25014
--HG--
extra : moz-landing-system : lando
The graph contains some extra things like toolchains, fetches and packaging
tasks that people will almost never want to run on their own. This change gets
them out of the default fuzzy selection interface, and makes it so --full is
needed to schedule them.
Differential Revision: https://phabricator.services.mozilla.com/D24187
--HG--
extra : moz-landing-system : lando
When we parse template arguments, we stuff them all in kwargs['templates'],
however we don't delete the old argument. This results in all kinds of unused
variables lying around in kwargs. E.g we would have both
kwargs['templates']['env'] and kwargs['env'] (the latter being unused). This is
the main reason why all the selector's run functions need to have a **kwargs at
the end of them.
Depends on D22022
Differential Revision: https://phabricator.services.mozilla.com/D22023
--HG--
extra : moz-landing-system : lando
This allows us to refactor mach_commands.py so we can call self.handle_presets
implicitly. This in turn gives us a future place to add shared code and makes
adding new selectors easier.
Differential Revision: https://phabricator.services.mozilla.com/D22021
--HG--
extra : moz-landing-system : lando
This creates a global preset file at:
tools/tryselect/try_presets.yml
Any presets defined here will be available for everyone to use.
Differential Revision: https://phabricator.services.mozilla.com/D21435
--HG--
extra : moz-landing-system : lando
This fixes a regression from bug 1483228. It started printing a message to
stdout whenever the local state dir was generated. This caused a failure in
these cramtests since they depend on the stdout of the shell process being
identical.
To fix, make sure we trigger state dir creation in the setup script and
redirect to /dev/null.
Differential Revision: https://phabricator.services.mozilla.com/D19155
--HG--
extra : moz-landing-system : lando
Passing in --exact reverses the behaviour of the ' operator. For example,
take the query "foo 'bar".
By default: foo is a fuzzy match and bar is an exact match.
With --exact: foo is an exact match and bar is a fuzzy match
Differential Revision: https://phabricator.services.mozilla.com/D16734
--HG--
extra : moz-landing-system : lando
Currently selectors that generate the taskgraph receive a list of task labels
back. While this works, it isn't very robust and limits the selectors in what
they can accomplish.
For example, it means we need to use regexes to map test suites to tasks. If
we had the full taskgraph, we could use filter functions instead.
There are also new selectors in the works which will need this (like ./mach
try chooser). Finally the try syntax selector would also need this if we
ever migrate it to use the 'try_task_config.json' format.
Differential Revision: https://phabricator.services.mozilla.com/D12570
--HG--
extra : moz-landing-system : lando
One of the big downsides to |mach try fuzzy| is that if you use the interactive
finder, there's no way to repeat your last action. If you want to run the same
push again, you have to manually re-select all the same tasks a second time.
It is possible to save presets, but this is fairly heavy-weight and (more)
permanent. Sometimes you just want to re-run a push a few times and forget
about it. It's also possible to craft the query on the command line with -q,
but then you don't get the immediate visual feedback, so you can't be sure that
you typed out the right things without actually pushing.
With |mach try again|, everytime you generate a try_task_config.json via
'fuzzy', 'empty' or any other subcommands that may exist in the future, it'll
get stored in a history file under ~/.mozbuild. Then running |mach try again|
will simply re-run the most recent try_task_config.json.
You'll also be able to view the whole history via |mach try again --list| and
select a specific try_task_config.json (i.e not the most recent one) via
|mach try again --index <index>|.
Example usage will be:
$ ./mach try fuzzy
<select a bunch of tasks>
$ ./mach try again
<re-pushes exact same set of tasks>
MozReview-Commit-ID: 3EZjVCy08uq
Depends on D1808.
Differential Revision: https://phabricator.services.mozilla.com/D1867
--HG--
extra : moz-landing-system : lando
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
Currently it's possible to specify a single query and take the union of terms with the '|'
symbol. However if you want to craft anything more complicated (i.e linux mochitest and
xpcshell, but windows reftest), it becomes really difficult. This allows developers to union
the result of multiple queries.
For example:
./mach try fuzzy -q "'linux 'mochitest | 'xpschell" -q "'windows 'reftest"
Differential Revision: https://phabricator.services.mozilla.com/D1838
Enables |./mach try fuzzy --talos-profile|. This template only applies to talos
tasks. It also provides --geckoProfile for consistency with |mach try syntax|,
but I don't like this name so it's hidden from the help.
The 'talos-profile.yml' template is also very specific (only applies to Talos
tasks). Ideally I'd like a general 'command.yml' template that just appends
arguments to the command for any arbitrary tasks. But then we'd need to invent
an expression syntax in try_task_config.json so we could make sure it only
applies to Talos. Then I thought rather than implement it for a specific
template, we should have a general way of doing this which could apply to any
and all of the templates.
Needless to say, it's a rabbit hole and something that's best left to a
follow-up so we don't delay this bug.
MozReview-Commit-ID: GhllZ7sr0ar
--HG--
extra : rebase_source : 1de4deecc2f73130904d7c95d4ff12f85883cd91
This enables the syntax like:
./mach try fuzzy dom/indexedDB
This will open up the fzf interface like normal, except only tasks
that have tests under dom/indexedDB will be selectable (and there
will only be one chunk per configuration).
This can be combined with -q/--query like normal:
./mach try fuzzy dom/indexedDB -q "!pgo !cov !asan"
When the tasks get scheduled, only the tests under the specified
path(s) will run within the harness.
MozReview-Commit-ID: IHRXXi5mB4G
--HG--
extra : rebase_source : 8a89f255591e6dfa31b1420196c4698f2015d10c
This sets the MOZHARNESS_TEST_PATHS environment variables for all tasks.
When specifying paths, this will cause many test tasks to only run the
tests under that directory as opposed to the normal default.
MozReview-Commit-ID: IHRXXi5mB4G
--HG--
extra : rebase_source : e7132311641f4d36a5bff56424988fbb3ae87238
This changes templates so they return an entire context dict instead of
only returning context based on their name. For example, now the 'path'
template can set context for 'env'.
A side effect of this is that there is no longer a 1-to-1 mapping of templates
in tryselect and taskgraph.
MozReview-Commit-ID: IHRXXi5mB4G
--HG--
extra : rebase_source : 4d7e398e60598a5de7961fb126f1d05a0b983681
This makes use of pytest's generation feature. To add a new
template test, just add a new entry containing the input and
expected output to the dict in test_templates.py
MozReview-Commit-ID: 4qMefYHMjAp
--HG--
extra : rebase_source : ba3049885d1a2485048e1ff9913be43317559376
