Instead of parsing the manifests and running the tests all in one go, this will
spawn an extra Firefox instance at the beginning that does nothing but parse the
manifest and dump them to a file.
This will allow the python harness to load and manipulate the test objects, before
sending them back to the JS harness as a list of tests to run. The main motivation
for this change is to implement run-by-manifest, a mode where we restart the
browser in between every test manifest. But there are other benefits as well, like
sharing the chunking logic used by other harnesses and the ability for the python
harness to stuff arbitrary metadata into the test objects.
For now, Android will continue to parse the manifests and run the tests all in one
go. Converting Android to this new mechanism will be left to a follow-up bug.
MozReview-Commit-ID: AfUBmQpx3Zz
--HG--
extra : rebase_source : 82e611e8795150daf01d791ea16517ec1ffb2896
Instead of parsing the manifests and running the tests all in one go, this will
spawn an extra Firefox instance at the beginning that does nothing but parse the
manifest and dump them to a file.
This will allow the python harness to load and manipulate the test objects, before
sending them back to the JS harness as a list of tests to run. The main motivation
for this change is to implement run-by-manifest, a mode where we restart the
browser in between every test manifest. But there are other benefits as well, like
sharing the chunking logic used by other harnesses and the ability for the python
harness to stuff arbitrary metadata into the test objects.
For now, Android will continue to parse the manifests and run the tests all in one
go. Converting Android to this new mechanism will be left to a follow-up bug.
MozReview-Commit-ID: AfUBmQpx3Zz
--HG--
extra : rebase_source : 955966c07bb650946c7c0e5706856f028335e850
Currently reftest.jsm uses logger.testEnd(..) to log all kinds of different
potential failures, from proper test failures to harness issues. This means
there are all kinds of edge cases that would cause multiple testEnd messages to
get logged, something the structured log protocol is supposed to prevent.
This modifies the reftest harness to instead use testStatus for everything. The
testEnd call will always be logged with status "OK". This required some changes
to the reftest formatter and reftest selftests.
MozReview-Commit-ID: 8RxsmHW49oy
--HG--
extra : rebase_source : 2f53e63f00bfc5e0e751b0f2bb3dc67477dc07eb
Non-ascii characters in error messages can cause exceptions when processing
structured log messages. The mochitest MessageLogger already handles such
cases; this is a simpler implementation for the reftest OutputHandler.
This makes the code work the way it looks like it was intended to work
based on what was already in output.py.
MozReview-Commit-ID: 8LJHxlfiQQ8
--HG--
extra : rebase_source : 46b2435ddbd3e8485536806d9fc7987599227f16
This allows the logs to work with the structured reftest viewer.
MozReview-Commit-ID: CY71vSdDjLP
--HG--
extra : rebase_source : 6b83d98aff1c5e73ac0a802b5a83b8be95adf59a
Ensure that the first line of the log for failing tests is identical to
one that would be produced by the tbplformatter from mozlog. This means
that treeherder will be able to sucessfully cross-match error lines.
Also make the data structures used for storing screenshots identical
between reftests and web-platform-tests so that we can start to move
toward removing the reftest-specific codepath here.
MozReview-Commit-ID: FZQXLjj9Ejv
--HG--
extra : rebase_source : 3e9bf70ff7915817b8c02032c8da0d0a659e4661
Structured logs bring many benefits. We can stop parsing the logs for magic strings, we
can modify the format without breaking things, and we can stream results into systems like
ActiveData. The structured logs originate primarily in reftest.js. StructuredLog.jsm is
used to generate the JSON-based log stream. Finally OutputHandler in the python harness
reads structured output from stdout, and formats it into human readable form.
MozReview-Commit-ID: G3ZLkMRl6p7
--HG--
extra : commitid : J3ui9XYWR3Q
extra : rebase_source : 77ed0ba842cc8e557141fb3494212b06868c728a
extra : amend_source : 735d48225a2e627e0fe45fc11b50b6c49a885a4b
extra : source : d1779fe421c3c7cd8e3d191816776390dc104f37
Structured logs bring many benefits. We can stop parsing the logs for magic strings, we
can modify the format without breaking things, and we can stream results into systems like
ActiveData. The structured logs originate primarily in reftest.js. StructuredLog.jsm is
used to generate the JSON-based log stream. Finally OutputHandler in the python harness
reads structured output from stdout, and formats it into human readable form.
--HG--
extra : commitid : J3ui9XYWR3Q
extra : rebase_source : 6bae978126dbd5beddc39332c7cbce0c1354cd87
extra : amend_source : 735d48225a2e627e0fe45fc11b50b6c49a885a4b
