This patch adds a new layer to mozperftest that allows people to run all the tests that produced an alert locally.
It does this by querying Treeherder for the alert information, and determining the tests to run from there. A `--alert-exact` option is added to enable users to run the exact same command as was run in CI as well. This is done by querying the task information of the alerting task, and parsing the options from it in a framework-specific manner.
All harnesses have option parsing implemented for them, however some of them, such as AWSY are more manual due to how the argument parser is implemented in that harness.
Differential Revision: https://phabricator.services.mozilla.com/D219965
This patch changes how the coveragerc file is handled between raptor, and mozperftest unit test runs. Instead of using a single file, this patch adds a second one that is specific to raptor, and renames the original to `.mpt-coveragerc`.
Differential Revision: https://phabricator.services.mozilla.com/D219964
This patch adds a new layer to mozperftest that allows people to run all the tests that produced an alert locally.
It does this by querying Treeherder for the alert information, and determining the tests to run from there. A `--alert-exact` option is added to enable users to run the exact same command as was run in CI as well. This is done by querying the task information of the alerting task, and parsing the options from it in a framework-specific manner.
All harnesses have option parsing implemented for them, however some of them, such as AWSY are more manual due to how the argument parser is implemented in that harness.
Differential Revision: https://phabricator.services.mozilla.com/D219965
This patch changes how the coveragerc file is handled between raptor, and mozperftest unit test runs. Instead of using a single file, this patch adds a second one that is specific to raptor, and renames the original to `.mpt-coveragerc`.
Differential Revision: https://phabricator.services.mozilla.com/D219964
This patch modifies the background-resource (formerly background-cpu) android test to collect resource information at distinct times rather than only at the end of the test run.
The collection script is split in two here for cpu, and memory. The python parsing script is also modified to be able to gather the resources from multiple files, and the name is changed to better reflect what it does.
This patch also renames the test to better reflect that it's a resource usage test and not just a cpu test.
Differential Revision: https://phabricator.services.mozilla.com/D217097
We can't use `shutil.copy_tree` here, since we need the `update=True`
functionality implemented by the `distutils` variant, but not by the
`shutil` one. This looks like the only place in the repo we use this
functionality, so it seems fine to implement an equivalent ourselves
here directly.
Differential Revision: https://phabricator.services.mozilla.com/D211632
We can't use `shutil.copy_tree` here, since we need the `update=True`
functionality implemented by the `distutils` variant, but not by the
`shutil` one. This looks like the only place in the repo we use this
functionality, so it seems fine to implement an equivalent ourselves
here directly.
Differential Revision: https://phabricator.services.mozilla.com/D211632
This patch adds a couple additional variable that shell scripts can use. One of them is the name of the app being tested, and the other is a temporary directory for the script that is removed once the script finishes. At the same time, the logging from the script is changed to use print directly instead of the logger due to bug 1900056.
Differential Revision: https://phabricator.services.mozilla.com/D212628
A regression was introduced in the startup tests accidentally when re-formatting the android_startup file.
This patch resolves that issue, which was forgetting a sleep timer of 4 seconds to allow the skip on-boarding flag to propagate
Differential Revision: https://phabricator.services.mozilla.com/D212645
This patch adds a new layer that will allow us to run custom/adhoc bash scripts in mozperftest. Currently, the layer can only handle bash scripts. At the same time, a new flavor is added for this, and additional script parsing capabilities are added.
Differential Revision: https://phabricator.services.mozilla.com/D211740
This patch adds a layer to gather the version of a given binary. The layer makes use of the tooling from `setups.py` to do this.
Differential Revision: https://phabricator.services.mozilla.com/D211739
This patch adds a layer to setup the binary, and provide a single interface for specifying it and using it through the metadata. At the same time, some changes were made to the perfherder metrics production to gather the application name, and version from the metadata. The intermediate results are also properly passed into the perfherder blob production step as they were previously being ignored.
Differential Revision: https://phabricator.services.mozilla.com/D211738
This patch adds a set of classes that will be used for gathering the binary path, and binary version in the VersionProducer, and BinarySetup layers.
Differential Revision: https://phabricator.services.mozilla.com/D211737
Currently, the browsertime used by mozperftest is outdated and
maintained separately from the browsertime used in raptor tests.
This patch upgrades the browsertime used in mozperftest by now sharing
the same browsertime that is used with raptors (in `src/tools/browsertime`).
Going forward we now only have to maintain one browsertime version. This
introduces the potential of more infra changes (because previously when
upgrading raptor's browsertime we only checked raptor tests, but now we
will have to verify mozperftest) but this should be worth it to stay up
to date.
Differential Revision: https://phabricator.services.mozilla.com/D208944
As a result of the mono-repo change we have a very easy way to run the startup tests on each commit vs on a nightly build.
This change allows us to push a try job, build the fenix/focus/geckoview_example apk and test the startup impact!
Differential Revision: https://phabricator.services.mozilla.com/D208240
As a result of the mono-repo change we have a very easy way to run the startup tests on each commit vs on a nightly build.
This change allows us to push a try job, build the fenix/focus/geckoview_example apk and test the startup impact!
Differential Revision: https://phabricator.services.mozilla.com/D208240
Webpagetest has modified the way that they accept keys, they used to accept it through a URL parameter, but now they require it to be in the header and with a different name
Differential Revision: https://phabricator.services.mozilla.com/D207159
This patch fixes a couple bugs with recording. The first is that the app wasn't being passed to mozproxy, and this was preventing the creation of the mitmproxy certificate for the app being used to record. The second is that the --proxy-deterministic flag isn't defined within the proxy layer even though it's used by it from the hooks.
Differential Revision: https://phabricator.services.mozilla.com/D199891
As the motorola G5 and the Pixel 2 have been removed from CI we are also going to remove references to them in our documentation and taskcluster code.
Differential Revision: https://phabricator.services.mozilla.com/D199294
The issue we were encountering here was a missing metric for canva.com intermittently occuring
That ocured because of the location of that metric(first contentful paint) in the returned json moved locations for just canva.com, and so canva.com was removed whilst the problem debugged. Afterwards it was discovered that this issue was covering up another intermittent about json decoding error which was in turn resolved by removing code which removed "www." from links used with python requests.
Differential Revision: https://phabricator.services.mozilla.com/D199238
This patch fixes the mozperftest docs to mention mochitest in the writing section, as well as fixes the `info` calls so that they mention the JSON.stringify requirement.
Differential Revision: https://phabricator.services.mozilla.com/D196853
This patch prevents the mozperftest unit tests from installing the opencv-python/mozperftest-tools module during unit tests.
Differential Revision: https://phabricator.services.mozilla.com/D195901
This patch updates mozperftest-tools to 0.2.9, and removes the opencv-python 4.5 install since it interferes with the new opencv-python package that has a lower minimum version for python (version 3.7).
Differential Revision: https://phabricator.services.mozilla.com/D195774
This patch fixes an issue where only `perftest.ini` manifests were being parsed for mozperftest tests in perfdocs. It's temporary as we will move all ini manifests to toml in the future. At the same time, this patch adds missing documentation for some new mozperftest tests that use a toml manifest to define the tests.
Differential Revision: https://phabricator.services.mozilla.com/D194251
These are a first-cut of service worker performance tests, just running as
vanilla mochitests for now. Bug 1832059 tracks the integration work necessary.
Differential Revision: https://phabricator.services.mozilla.com/D177427
Greg Mierzwinski