This just adds two basic tests, one for a passing test and another for a
failing one. In mochitest, we use privileged APIs to also tests crashes,
assertions, asan and leaks. But these APIs aren't available to reftests
so I'm not sure how we can test these things.
I figure it's not worth holding the framework up on this though, I'll file
a follow-up to figure out something to do for that.
MozReview-Commit-ID: 59TSbsugT5T
--HG--
extra : rebase_source : 72ecd817017c8b7d55eab879db4f6ad5fecc54c0
Skip tests that are expected to fail in both Stylo and Gecko modes. They would unexpectedly "pass" in styloVsGecko mode when comparing the two failures, which is not a useful result.
MozReview-Commit-ID: 3mOpjU225Q1
--HG--
extra : rebase_source : 22bb5d4e3c5138ef832995eaf5716824f4707ffe
extra : source : d40fb20c9a49d0797c0eeae613a04912b12a28f7
This renames killNamedOrphans to killNamedProc and removes the check for
parent proc id, so that any xpcshell or ssltunnel process is killed before
starting a mochitest or reftest run. For reftests, this is moved out of
the desktop harness and into the remote/android harness, since desktop
reftests do not use xpcshell or ssltunnel.
Since these tests harnesses are already disabling Safe Browsing and tracking
protection, they should also disable the other two features that cause
lists to be downloaded from the Mozilla shavar server.
MozReview-Commit-ID: 2158qRU4XZx
--HG--
extra : rebase_source : 287bfb7580538cca7f58ab035b7141166346c318
The FrameLoaderOwner interface has been implemented in WebIDL for several
years now, so these QIs are simply unnecessary overhead.
MozReview-Commit-ID: LAzvfm5Qhy0
--HG--
extra : rebase_source : 2495c07df21c474f5fabc257ff4db43b0d8047e4
The login reputation checks depend on a server lookup and therefore would
render non-deterministic the performance and correctness of tests.
MozReview-Commit-ID: Bil0rSZsGPT
--HG--
extra : rebase_source : ae06a028c71eee323307ecd4e62bbf1e8a14fe13
The login reputation checks depend on a server lookup and therefore would
render non-deterministic the performance and correctness of tests.
MozReview-Commit-ID: Bil0rSZsGPT
--HG--
extra : rebase_source : c9a641e4bf49c48bf864ed546bf2ae6eb51c27e4
This shouldn't be at module level.
This isn't technically needed for this series. I caught it during
an earlier attempt at teasing out all Python import dependencies
to run `mach`. Why let a good patch go to waste.
MozReview-Commit-ID: FwwdZqcKtpq
--HG--
extra : rebase_source : a8ce2cf452dc8c03371dc384ea4f1be9195b5d30
This also guards against passing a non-test parameter to reftest.startAfter.
MozReview-Commit-ID: FoqqN4D7sv7
--HG--
extra : rebase_source : 0e5d879467c04803e3ac565b979aceaad3176597
This avoids mistakenly blaming the first reftest in the suite for a crash that
happens during startup. Blaming that reftest can result in fragmentation of the
resulting issues because any bugs filed by sheriffs will be test-specific instead
of grouped together into a single bug.
MozReview-Commit-ID: K4YelNzXxzg
--HG--
extra : rebase_source : b291f365956997bbc3d591addee6817ad3e0ff9b
Test harnesses may use STYLO_FORCE_ENABLED, so we need to check this value when
building the reftest conditions.
MozReview-Commit-ID: 998UMZHNKLl
--HG--
extra : rebase_source : c52e2539c0384c2c300b729ac88fe3381e127509
This adds commands to start a reftest session, run a test, and end the
session. It as assumed that after you start a reftest session you will
just run reftests until you end the session. When starting a session
the user provides a string indicating when screenshots should be
taken, and an object mapping urls to a count of the number of times
that url is expected to be used in the session, to help with
caching. Running the tests takes a url to a test, an expected status,
a timeout, and a nested list of possible references, in which each
entry at a specific level is combined by OR and nested references are
combined by AND.
The implementation is heavilly inspired by the existing reftest
harness, starting a minimal window with no tabs, and loading the urls
directly in there. In order to get a screenshot in the e10s case we
have to pass the DRAW_VIEW and USE_WIDGET_LAYERS flags when taking the
screenshot.
For performance we heavily cache canvases; for references that will be
repeated we cache the full canvas with image, and we also cache a
single canvas to use for all other screenshots to avoid the overhead
of repeatedly creating a new canvas element.
MozReview-Commit-ID: JOFvtmH7tg
This adds commands to start a reftest session, run a test, and end the
session. It as assumed that after you start a reftest session you will
just run reftests until you end the session. When starting a session
the user provides a string indicating when screenshots should be
taken, and an object mapping urls to a count of the number of times
that url is expected to be used in the session, to help with
caching. Running the tests takes a url to a test, an expected status,
a timeout, and a nested list of possible references, in which each
entry at a specific level is combined by OR and nested references are
combined by AND.
The implementation is heavilly inspired by the existing reftest
harness, starting a minimal window with no tabs, and loading the urls
directly in there. In order to get a screenshot in the e10s case we
have to pass the DRAW_VIEW and USE_WIDGET_LAYERS flags when taking the
screenshot.
For performance we heavily cache canvases; for references that will be
repeated we cache the full canvas with image, and we also cache a
single canvas to use for all other screenshots to avoid the overhead
of repeatedly creating a new canvas element.
MozReview-Commit-ID: JOFvtmH7tg
--HG--
extra : rebase_source : ab5a2ef2e450b9bbdc6bc3c9487ed5dfda2c1d4b
This is needed before we can upgrade to flake8 3.3.0, as that version starts flagging these errors.
These files were modified by running:
autopep8 --select E305 --in-place -r <dir>
on the affected directories. I did it one dir at a time and verified the result after each.
MozReview-Commit-ID: FmlsfiKIbtr
--HG--
extra : rebase_source : 9df32258cadff5d27a0e72113c57f782756c0b18
