Use finer granularity for some reftest/mochitest SCHEDULES.exclusive entries,
so that reftest-plain does not run when only crashtests are modified, and
vice versa; similarly, break up mochitest into mochitest/browser-chrome/chrome/
a11y. Use schedules-component instead of category.
Differential Revision: https://phabricator.services.mozilla.com/D60085
--HG--
extra : moz-landing-system : lando
Remove the remainder of automation.py.in and the build support for generating
automation.py.
Some of this functionality was in use, especially for android tests.
Some code was moved or re-implemented in remoteautomation.py or in the affected
harness(es). Some features were removed: There are some minor changes in behavior.
For instance, instead of using a different server startup timeout for debug builds,
one value is used for all builds (due to performance improvements over time,
the longer timeout is no longer needed).
Differential Revision: https://phabricator.services.mozilla.com/D66839
--HG--
extra : moz-landing-system : lando
These have been broken for a while, because we don't run them on automation...
The way we import PDFJS in reftest.jsm was broken. And PDFJS updates caused
breakage.
Differential Revision: https://phabricator.services.mozilla.com/D66669
--HG--
extra : moz-landing-system : lando
Remove unused code from automation.py.in, move some android-specific code from
automation.py.in to remoteautomation.py, and eliminate some other easily-replaced
code. In the long term we want to eliminate automation.py.in completely; I may
attempt that once these changes have landed.
Differential Revision: https://phabricator.services.mozilla.com/D66306
--HG--
extra : moz-landing-system : lando
Currently the RefTest manifest parser has 5 possible statuses:
0 => EXPECTED_PASS
1 => EXPECTED_FAIL
2 => EXPECTED_RANDOM
3 => EXPECTED_DEATH (aka skip)
4 => EXPECTED_FUZZY
In the manifests, the last status annotation that appears on the line will take
precedence. For example:
skip-if(true) fails-if(true) == test1.html ref.html
fails-if(true) skip-if(true) == test2.html ref.html
The first test will have an expected status equal to EXPECTED_FAIL, whereas the
second one will be EXPECTED_DEATH. The same holds true for any combination of
'fail/random/skip/fuzzy' annotations. This means developers need to be very
careful about the order they specify these annotations as getting the order
wrong can easily lead to unexpected behaviour.
With the introduction of defaults in bug 1616368, the risk of unexpected behaviour
is far greater. Since defaults are simply prepended to the test line, a manifest
that looks like:
defaults skip-if(true)
== test1.html ref.html
fails-if(true) == test2.html ref.html
will actually skip the first test, but run the second (since the fails-if
overwrites EXPECTED_DEATH with EXPECTED_FAIL).
The root of the problem appears to be that 'skip' and 'fuzzy' are not actually
test statuses. They are modifiers that affect how we run the test, but don't
actually affect whether the test is expected to pass or fail.
Therefore, this patch solves the problem by making 'skip/skip-if' its own thing
that does not get overwritten by other failure types. In otherwords, a 'skip-if'
can appear before or after a 'fails-if' and it will have the same meaning.
Differential Revision: https://phabricator.services.mozilla.com/D64457
--HG--
extra : moz-landing-system : lando
Reftest-family test suites can be "run-by-manifest" (reftest-plain, on desktop), or not
(all reftest-family suites on Android, crashtest and jsreftest on desktop). When not
run-by-manifest, code in reftest.jsm is used to format the suite_start data; when
run-by-manifest, code in runreftest.py is used. Currently, the reftest.jsm implementation
submits a simple array of test IDs -- no associated manifests -- possibly consistent with
the concept that tests are not being run/grouped by manifest.
This patch updates the reftest.jsm implementation to generate the same format of
suite_start data as the python implementation: An array of dictionaries mapping
manifest IDs to arrays of associated test IDs.
Differential Revision: https://phabricator.services.mozilla.com/D63875
--HG--
extra : moz-landing-system : lando
reftest-plain test identifiers may be tuples like ['this', '==', 'that'], which are
undesirable to ActiveData. This patch converts such identifiers to simple strings,
'this == that' in the suite_start message (string format is already used in other
messages, like test_start).
Differential Revision: https://phabricator.services.mozilla.com/D63976
--HG--
extra : moz-landing-system : lando
Adds some tests to cover error cases in reftest manifests. This is far from
comprehensive, but gives us an easy place to add more in the future.
Differential Revision: https://phabricator.services.mozilla.com/D63718
--HG--
rename : layout/tools/reftest/selftest/files/defaults-invalid.list => layout/tools/reftest/selftest/files/invalid-defaults.list
extra : moz-landing-system : lando
Currently a parsing error causes reftest.jsm to hang, leaving the browser open.
Once the browser is manually closed, the Python side then emits an unrelated
traceback (since it assumes that parsing was succesful).
This ensures we both quit Firefox and check that parsing was successful (and
exit gracefully if not).
Differential Revision: https://phabricator.services.mozilla.com/D63728
--HG--
extra : moz-landing-system : lando
With the new 'defaults' key being added, 'default-preferences' is now
redundant. This commit converts all existing uses of 'default-preferences' to
use 'defaults' instead.
Differential Revision: https://phabricator.services.mozilla.com/D63247
--HG--
extra : moz-landing-system : lando
This introduces a new 'defaults' key to reftest manifests. It works similarly
to the existing 'default-preferences' key except it can be used for any test
modifier (e.g, skip-if, fail-if, pref, etc.).
See the changes to README.txt in this commit for an explanation on how to use
it.
Differential Revision: https://phabricator.services.mozilla.com/D63246
--HG--
extra : moz-landing-system : lando
Add text with the total number of pixels that differ and the maximum
difference per channel, since this is useful when generating fuzzy
annotations for e.g. antialiasing differences.
Differential Revision: https://phabricator.services.mozilla.com/D62566
--HG--
extra : moz-landing-system : lando
Braces on switch statement cases so scopes are limited.
A missing semicolon.
A mixup with infoStrings/errorStrings return value for UpdateLayerTree message.
Differential Revision: https://phabricator.services.mozilla.com/D62406
--HG--
extra : moz-landing-system : lando
Use the manifest ID rather than the manifest path in the reftest suite_start message.
Differential Revision: https://phabricator.services.mozilla.com/D62081
--HG--
extra : moz-landing-system : lando
Both _setup_objdir() and run_android_test() have code to set the
default test path, and _setup_objdir()'s crashtest path is wrong;
let's rely on the correct code in run_android_test() instead.
Differential Revision: https://phabricator.services.mozilla.com/D61946
--HG--
extra : moz-landing-system : lando
This patch was generated with a script. It doesn't include all files:
- Files that use the preprocessor or fail to parse are skipped
- Files that are loaded as JSMs but don't use the .jsm extension are skipped (those will be renamed in Bug 1609269)
It was generated with the following command using d855222aa2/no-this-property-read.js:
```
hg revert --all &&
cp .gitignore .rgignore &&
rg --files-without-match -g '*.jsm' '^#endif|^#include|^#filter' | jscodeshift --stdin --transform ~/Code/jsm-rewrites/no-this-property-read.js --ignore-pattern ./mobile/android/modules/Sanitizer.jsm --ignore-pattern ./js/xpconnect/tests/unit/syntax_error.jsm &&
./mach eslint `hg st | rg '^M ' | sed 's/^M //'`
```
Differential Revision: https://phabricator.services.mozilla.com/D60187
--HG--
extra : moz-landing-system : lando
Use test identifiers that look like posix-compliant relative file paths rather than URLs; identifier "paths" are relative to topsrcdir -- an intuitive, brief mapping from identifier to test file.
This patch adds (expands) a --topsrcdir option to runreftest.py and updates CI configuration to set it appropriately. runreftest.py calculates a manifest ID, a path relative to topsrcdir, for each requested manifest and passes those manifest IDs to manifest.jsm. In manifest.jsm, a new manifest ID is generated when a manifest is included from a new directory. Test identifiers are generated by adding the test file name to the manifest ID.
Examples (url -> identifier):
- file:///builds/worker/workspace/build/tests/reftest/tests/dom/svg/crashtests/398926-fill.svg -> dom/svg/crashtests/398926-fill.svg
- file:///Z:/task_1577755655/build/tests/reftest/tests/dom/svg/crashtests/398926-fill.svg -> dom/svg/crashtests/398926-fill.svg
- http://10.7.205.207:8854/tests/dom/svg/crashtests/398926-fill.svg -> dom/svg/crashtests/398926-fill.svg
- file:///builds/worker/workspace/build/tests/jsreftest/tests/js/src/tests/jsreftest.html?test=test262/built-ins/Array/prototype/indexOf/15.4.4.14-9-b-i-31.js -> js/src/tests/test262/built-ins/Array/prototype/indexOf/15.4.4.14-9-b-i-31.js
- chrome://reftest/content/bugs/299837-2.xhtml -> layout/reftests/bugs/299837-2.xhtml
- http://localhost:36067/1577754972876/1/289480.html#top -> layout/reftests/bugs/289480.html#top
- about:blank -> about:blank
- data:text/html,<body> -> data:text/html,<body>
Differential Revision: https://phabricator.services.mozilla.com/D58415
--HG--
extra : moz-landing-system : lando
We enable the pref in the harness so that local testing with --enable-fission will get the same results as treeherder.
Differential Revision: https://phabricator.services.mozilla.com/D56313
--HG--
extra : moz-landing-system : lando
