Various code in the Raptor webextension that is written
asynchronously isn't used with 'await' or at least a
Promise chain. As result there will be overlapping
attempts to send messages to the control server that
can even vary in their order, and causing tests
to fail.
With that patch the amount of extra console logging
will be reduced to only those cases where sending the
message to the control server failed.
Differential Revision: https://phabricator.services.mozilla.com/D64348
--HG--
extra : moz-landing-system : lando
This patch adds some stability and comparison optimizations to the Browsertime arguments. It adds a viewport setting to ensure that all browsers we will test on use the same browser size (only affects desktop). The pageLoadStrategy is also set to `none` so that we can use the `pageCompleteCheckStartWait` and `pageCompleteCheckPollTimeout` options. Those options determine how much Selenium polls (and when it starts polling) to check if there are results available.
The size of the browser is also set to a specific size to prevent regressions/improvements (alerts) from occuring when we update Browsertime and the default screen resolution changes. This patch also disables the Browsertime Firefox extension since we don't need it.
Differential Revision: https://phabricator.services.mozilla.com/D63182
--HG--
extra : moz-landing-system : lando
This patch adds all Fenix pageload tests to raptor-browsertime. To handle how the task is specified (specifying run-on-projects by raptor-test) the raptor.py taskgraph transforms had to be modified.
The transform modifications change a couple things. First, the `raptor-subtests` is now the first thing to have it's keyed-by entries handled because it's values are used as keyed-by entries on other fields such as run-on-projects. Once it's values are resolved, we immediately split/clone the task into one per subtest entry (adding a `chunk-number` field that is used for more setup in the next transforms). After this, the rest of the fields have their keyed-by entries resolved.
Differential Revision: https://phabricator.services.mozilla.com/D63222
--HG--
extra : moz-landing-system : lando
This patch allows us to use the enable/disable charging commands for power tests, and moves the android power tests onto the primary perf bitbar devices. It also removes the use of the --host argument that was previously required to run power tests. Disabling charging occurs when the device is being setup, and when the device is being setup within raptor (to accomadate different entry points). Enabling charging occurs when the mach command finishes, or during the clean_up stage in raptor in normal, passing executions. In case any errors occur within raptor, and due to the multiple entry points, charging is enabled where those errors are caught as well.
Differential Revision: https://phabricator.services.mozilla.com/D42373
--HG--
extra : moz-landing-system : lando
This patch upgrades the major browsertime version used in-tree from 4 to 8 (including some additional fixes to fix some failing tests on our end).
We also add the node v10 requirement in this patch. Also, there were some changes in the browsertime repo's visualmetrics.py script that made it necessary to change where we find the file.
Differential Revision: https://phabricator.services.mozilla.com/D59235
--HG--
extra : moz-landing-system : lando
This patch adds the initial set of documentation that will be added to the source-docs website.
Differential Revision: https://phabricator.services.mozilla.com/D60241
--HG--
extra : moz-landing-system : lando
This patch adds the PerfDocs Generator. Some modificiations to `run_perfdocs` were also made because verification always has to occur before the documentation generation, and generation always has to be run after verification (if verification passes).
Differential Revision: https://phabricator.services.mozilla.com/D57111
--HG--
extra : moz-landing-system : lando
Raptor was previously generating two artifacts for the `run-visual-metrics.py`
script to consume: `jobs.json` and `application.json`. These artifacts have
been merged.
Differential Revision: https://phabricator.services.mozilla.com/D62363
--HG--
extra : moz-landing-system : lando
The `run-visual-metrics.py` script was intended to consume a `jobs.json` file
containing one `browsertime.json` per video. However it was not being used as
such and was continuously re-processing the first video specified in the
`browsertime.json` file. If a job were submitted with a `browsertime.json`
containing 15 videos and 15 different videos, only the first would be
processed. This leads to us having incorrect metrics because over all runs all
the metrics will be identical.
Now we only specify the `browsertime.json` in the `jobs.json` file and extract
the paths to videos from there. Also because we are never downloading inputs
this way, we get to remove some dead code and our dependency on `requests`.
Differential Revision: https://phabricator.services.mozilla.com/D62320
--HG--
extra : moz-landing-system : lando
With this patch, we now remove all blank lines from the profiles we produce which cause Browsertime 8 to break.
Differential Revision: https://phabricator.services.mozilla.com/D62370
--HG--
extra : moz-landing-system : lando
Stop using mozcrash.check_for_java_exception(), which scans logcat for uncaught
exceptions. Instead, check for java_stack in the extra dump information during
normal crash reporting; if java_stack is present, display the java exception and
stack instead of dumping a full native crash report.
Differential Revision: https://phabricator.services.mozilla.com/D61836
--HG--
extra : moz-landing-system : lando
This patch will check the exit code of the ProcessHandler() that
calls the tooltool script, and will also retry when it fails.
Differential Revision: https://phabricator.services.mozilla.com/D60397
--HG--
extra : moz-landing-system : lando
Stylebench is patterned after Speedometer and calculates overall score from subtests in the same way. Stylebench should be marked as subtest_lower_is_better just like Speedometer.
Differential Revision: https://phabricator.services.mozilla.com/D56557
--HG--
extra : moz-landing-system : lando
