This adds commands to start a reftest session, run a test, and end the
session. It as assumed that after you start a reftest session you will
just run reftests until you end the session. When starting a session
the user provides a string indicating when screenshots should be
taken, and an object mapping urls to a count of the number of times
that url is expected to be used in the session, to help with
caching. Running the tests takes a url to a test, an expected status,
a timeout, and a nested list of possible references, in which each
entry at a specific level is combined by OR and nested references are
combined by AND.
The implementation is heavilly inspired by the existing reftest
harness, starting a minimal window with no tabs, and loading the urls
directly in there. In order to get a screenshot in the e10s case we
have to pass the DRAW_VIEW and USE_WIDGET_LAYERS flags when taking the
screenshot.
For performance we heavily cache canvases; for references that will be
repeated we cache the full canvas with image, and we also cache a
single canvas to use for all other screenshots to avoid the overhead
of repeatedly creating a new canvas element.
MozReview-Commit-ID: JOFvtmH7tg
--HG--
extra : rebase_source : ab5a2ef2e450b9bbdc6bc3c9487ed5dfda2c1d4b
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
--HG--
extra : rebase_source : 9df32258cadff5d27a0e72113c57f782756c0b18
This won't be an issue until the 57 release but at that time, the
reftest extensions will only load if the extensions.legacy.enabled
preference is set to true and Cu.isInAutomation, which depends on
another preference, is true. Set the appropriate prefs here for reftests.
MozReview-Commit-ID: JUWCPTsuJU
--HG--
extra : rebase_source : 042a5696d1926e7ca434a7c4213395176a080549
This catches a common problem where somebody adds a fuzzy annotation on
a test to work around some minor differences. Later the differences go
away, but since the test harness doesn't catch that, nobody is the
wiser. Subsequently a "real" regression can reintroduce differences
which are hidden by the stale fuzzy annotations.
With this patch, if the annotations are set up properly, the test
harness will flag tests as "UNEXPECTED-PASS" when the differences go
away. This will require the patch author to reduce the allowed fuzziness
parameters, and will make it easier to catch subsequent regressions.
MozReview-Commit-ID: B3rGPFLXkCu
This patch extends the "fuzzy" and "fuzzy-if" annotations so that they
accept a range of fuzziness values, much like the "asserts" and
"asserts-if" annotations. If the test produces differences that are
within the specified range, the test is considered as passing. Any
differences outside the ranges are considered failures. If the test is
marked as a "!=" test, then the opposite is true.
If the range given to "fuzzy" or "fuzzy-if" is a single value, it is
interpreted as being a range from zero to the specified value. This is
in contrast to "asserts" and "asserts-if" which match exactly the value
specified. The reason for this is mostly for backwards compatibility
with existing reftest annotation semantics.
MozReview-Commit-ID: 6qUU6FQ5mYP
This simply augments the logging in the reftest harness so that if a reftest
passes due to fuzzy matching, it reports both the actual fuzz numbers and the
maximum allowed fuzz numbers.
MozReview-Commit-ID: G356vBRXYUc
Remove the forbiddenURI pref which was removed in bug 1274893 as well
as browser.safebrowsing.enabled which got renamed in bug 1025965.
Set dummy URLs for all of the network endpoints.
MozReview-Commit-ID: Efk2fv6cC3g
--HG--
extra : rebase_source : 9fbb3eb0fa7f002fe24577a8a0870ec4d1b7cf31
Our linux32-debug build is very slow to startup when running mochitests on aws.
Sometimes we see similar behavior on other linux platforms. Intermittently,
in this environment, startup takes longer than the 120 seconds that marionette
waits, resulting in test failures in bug 1261598. Increasing the marionette
startup timeout to 180 seconds appears to effectively avoid these failures.
Remove suggested and enhanced tiles along with related campaign, frequency-cap, inadjacency, pings, preferences, strings, styles, tests.
MozReview-Commit-ID: FkjaSpSFQHu
--HG--
extra : rebase_source : 1c58ac542180f0abb290639ec1c61b9edf3d0a51
This prefix can't be produced by the standard structured logging
formatters because it doesn't know that the tests are
reftests. Therefore the reftest harness has a hack to add this prefix,
and the unstructured analyzer doesn't work with wpt reftests. This can
simply be solved by making the prefix optional, but looking for the
"INFO - " prefix that mozharness adds, when present, to identify the
start of the TEST- string.
MozReview-Commit-ID: GxWVWOSkAUZ
--HG--
extra : rebase_source : fbd8afa860da0fba4925b6bf2fc715cbf6c2688c
Non-ascii characters in error messages can cause exceptions when processing
structured log messages. The mochitest MessageLogger already handles such
cases; this is a simpler implementation for the reftest OutputHandler.
Previously the harness would check for navigator.plugins[Test Plug-in] in the browser window, but that is now blocked from working. Instead we can simply use an existing getTestPlugin function that enumerates the plugin through nsIPluginHost
MozReview-Commit-ID: Kp48u5iFkSa
Continue to allow non-multiprocessCompatible extensions in automation.
There are a ton of places that would need to be changed, many of which
will be changing soon anyway with the non-webextensions change in 57
so this is mostly the expedient route to keeping the tree green.
MozReview-Commit-ID: EZZoDVdhLfy
--HG--
extra : rebase_source : f83472bc1c88dd0deadbe485d9002499027ff07f
Continue to allow non-multiprocessCompatible extensions in automation.
There are a ton of places that would need to be changed, many of which
will be changing soon anyway with the non-webextensions change in 57
so this is mostly the expedient route to keeping the tree green.
MozReview-Commit-ID: EZZoDVdhLfy
--HG--
extra : rebase_source : 34aa762917566b052ade6372280caed72fbfbe9a
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : 81ba8bfee0ca96979cf8e30d75cdd47f06bc10ea
This patch does a few things:
a) Adds the resources location from the .app directory to the read whitelist
b) When it's a non-packaged build, mach run (and various mach tests) set an environment variable for the repo location which we allow reads from.
r=haik,froydnj
MozReview-Commit-ID: KNvAoUs5Ati
--HG--
extra : rebase_source : f637acff32fc8582732de932503dd696abc57877
This eliminates a 2 minute timeout seen at the end of Android mochitests
and reftests. Attempts to shutdown the web server were failing because
they were directed at IP 10.0.2.2 -- the loopback address for the
Android emulator.
The devicemanager killProcess() is updated to use force-stop first, then
use kill if force-stop does not work.
Browser test harnesses are updated to check if killProcess() worked, and
warn if it failed.
