This patch fixes an issue where the suite-level description error wasn't being output. Instead of using the manifest from the yaml for this error, we'll output the failure to all changed files. Some additional changes are made for mozperftest to handle tests with periods in their names.
Differential Revision: https://phabricator.services.mozilla.com/D227378
This patch introduces performance tests to evaluate the video encoding
performance in two usage scenarios: *Realtime* and *Record*. Each
scenario assesses the encoding using two key metrics:
1. Frame Encoding Latency: The time from encoding a single frame to
receiving its encoded result
2. Total Encoding Time: The duration from the first encoding request to
the reception of the last encoded result
In the *Realtime* scenario, minimizing the latency for each frame is
crucial. This metric is essential for applications like live streaming
and video conferencing, where timely delivery of each frame impacts user
experience.
In contrast, the *Record* scenarios prioritizes the total encoding time.
Here, the focus is on processing the entire sequence efficiently to
produce the final output.
By adding these tests, we aim to thoroughly assess and optimize the
encoder's performance under both scenarios, ensuring Gecko provides the
experience users expect.
Differential Revision: https://phabricator.services.mozilla.com/D224999
This patch adds some documentation about the mach perftest alert layer, and how to use it. This includes the additional options that it has.
Differential Revision: https://phabricator.services.mozilla.com/D224006
This patch adds a new layer to mozperftest that allows people to run all the tests that produced an alert locally.
It does this by querying Treeherder for the alert information, and determining the tests to run from there. A `--alert-exact` option is added to enable users to run the exact same command as was run in CI as well. This is done by querying the task information of the alerting task, and parsing the options from it in a framework-specific manner.
All harnesses have option parsing implemented for them, however some of them, such as AWSY are more manual due to how the argument parser is implemented in that harness.
Differential Revision: https://phabricator.services.mozilla.com/D219965
This patch changes how the coveragerc file is handled between raptor, and mozperftest unit test runs. Instead of using a single file, this patch adds a second one that is specific to raptor, and renames the original to `.mpt-coveragerc`.
Differential Revision: https://phabricator.services.mozilla.com/D219964
This patch adds a new layer to mozperftest that allows people to run all the tests that produced an alert locally.
It does this by querying Treeherder for the alert information, and determining the tests to run from there. A `--alert-exact` option is added to enable users to run the exact same command as was run in CI as well. This is done by querying the task information of the alerting task, and parsing the options from it in a framework-specific manner.
All harnesses have option parsing implemented for them, however some of them, such as AWSY are more manual due to how the argument parser is implemented in that harness.
Differential Revision: https://phabricator.services.mozilla.com/D219965
This patch changes how the coveragerc file is handled between raptor, and mozperftest unit test runs. Instead of using a single file, this patch adds a second one that is specific to raptor, and renames the original to `.mpt-coveragerc`.
Differential Revision: https://phabricator.services.mozilla.com/D219964
This patch modifies the background-resource (formerly background-cpu) android test to collect resource information at distinct times rather than only at the end of the test run.
The collection script is split in two here for cpu, and memory. The python parsing script is also modified to be able to gather the resources from multiple files, and the name is changed to better reflect what it does.
This patch also renames the test to better reflect that it's a resource usage test and not just a cpu test.
Differential Revision: https://phabricator.services.mozilla.com/D217097
We can't use `shutil.copy_tree` here, since we need the `update=True`
functionality implemented by the `distutils` variant, but not by the
`shutil` one. This looks like the only place in the repo we use this
functionality, so it seems fine to implement an equivalent ourselves
here directly.
Differential Revision: https://phabricator.services.mozilla.com/D211632
We can't use `shutil.copy_tree` here, since we need the `update=True`
functionality implemented by the `distutils` variant, but not by the
`shutil` one. This looks like the only place in the repo we use this
functionality, so it seems fine to implement an equivalent ourselves
here directly.
Differential Revision: https://phabricator.services.mozilla.com/D211632
This patch adds a couple additional variable that shell scripts can use. One of them is the name of the app being tested, and the other is a temporary directory for the script that is removed once the script finishes. At the same time, the logging from the script is changed to use print directly instead of the logger due to bug 1900056.
Differential Revision: https://phabricator.services.mozilla.com/D212628
A regression was introduced in the startup tests accidentally when re-formatting the android_startup file.
This patch resolves that issue, which was forgetting a sleep timer of 4 seconds to allow the skip on-boarding flag to propagate
Differential Revision: https://phabricator.services.mozilla.com/D212645
This patch adds a new layer that will allow us to run custom/adhoc bash scripts in mozperftest. Currently, the layer can only handle bash scripts. At the same time, a new flavor is added for this, and additional script parsing capabilities are added.
Differential Revision: https://phabricator.services.mozilla.com/D211740
This patch adds a layer to gather the version of a given binary. The layer makes use of the tooling from `setups.py` to do this.
Differential Revision: https://phabricator.services.mozilla.com/D211739
This patch adds a layer to setup the binary, and provide a single interface for specifying it and using it through the metadata. At the same time, some changes were made to the perfherder metrics production to gather the application name, and version from the metadata. The intermediate results are also properly passed into the perfherder blob production step as they were previously being ignored.
Differential Revision: https://phabricator.services.mozilla.com/D211738
This patch adds a set of classes that will be used for gathering the binary path, and binary version in the VersionProducer, and BinarySetup layers.
Differential Revision: https://phabricator.services.mozilla.com/D211737
Currently, the browsertime used by mozperftest is outdated and
maintained separately from the browsertime used in raptor tests.
This patch upgrades the browsertime used in mozperftest by now sharing
the same browsertime that is used with raptors (in `src/tools/browsertime`).
Going forward we now only have to maintain one browsertime version. This
introduces the potential of more infra changes (because previously when
upgrading raptor's browsertime we only checked raptor tests, but now we
will have to verify mozperftest) but this should be worth it to stay up
to date.
Differential Revision: https://phabricator.services.mozilla.com/D208944
As a result of the mono-repo change we have a very easy way to run the startup tests on each commit vs on a nightly build.
This change allows us to push a try job, build the fenix/focus/geckoview_example apk and test the startup impact!
Differential Revision: https://phabricator.services.mozilla.com/D208240
As a result of the mono-repo change we have a very easy way to run the startup tests on each commit vs on a nightly build.
This change allows us to push a try job, build the fenix/focus/geckoview_example apk and test the startup impact!
Differential Revision: https://phabricator.services.mozilla.com/D208240
Webpagetest has modified the way that they accept keys, they used to accept it through a URL parameter, but now they require it to be in the header and with a different name
Differential Revision: https://phabricator.services.mozilla.com/D207159
