Currently, the browsertime used by mozperftest is outdated and
maintained separately from the browsertime used in raptor tests.
This patch upgrades the browsertime used in mozperftest by now sharing
the same browsertime that is used with raptors (in `src/tools/browsertime`).
Going forward we now only have to maintain one browsertime version. This
introduces the potential of more infra changes (because previously when
upgrading raptor's browsertime we only checked raptor tests, but now we
will have to verify mozperftest) but this should be worth it to stay up
to date.
Differential Revision: https://phabricator.services.mozilla.com/D208944
As a result of the mono-repo change we have a very easy way to run the startup tests on each commit vs on a nightly build.
This change allows us to push a try job, build the fenix/focus/geckoview_example apk and test the startup impact!
Differential Revision: https://phabricator.services.mozilla.com/D208240
As a result of the mono-repo change we have a very easy way to run the startup tests on each commit vs on a nightly build.
This change allows us to push a try job, build the fenix/focus/geckoview_example apk and test the startup impact!
Differential Revision: https://phabricator.services.mozilla.com/D208240
Webpagetest has modified the way that they accept keys, they used to accept it through a URL parameter, but now they require it to be in the header and with a different name
Differential Revision: https://phabricator.services.mozilla.com/D207159
This patch fixes a couple bugs with recording. The first is that the app wasn't being passed to mozproxy, and this was preventing the creation of the mitmproxy certificate for the app being used to record. The second is that the --proxy-deterministic flag isn't defined within the proxy layer even though it's used by it from the hooks.
Differential Revision: https://phabricator.services.mozilla.com/D199891
As the motorola G5 and the Pixel 2 have been removed from CI we are also going to remove references to them in our documentation and taskcluster code.
Differential Revision: https://phabricator.services.mozilla.com/D199294
The issue we were encountering here was a missing metric for canva.com intermittently occuring
That ocured because of the location of that metric(first contentful paint) in the returned json moved locations for just canva.com, and so canva.com was removed whilst the problem debugged. Afterwards it was discovered that this issue was covering up another intermittent about json decoding error which was in turn resolved by removing code which removed "www." from links used with python requests.
Differential Revision: https://phabricator.services.mozilla.com/D199238
This patch fixes the mozperftest docs to mention mochitest in the writing section, as well as fixes the `info` calls so that they mention the JSON.stringify requirement.
Differential Revision: https://phabricator.services.mozilla.com/D196853
This patch prevents the mozperftest unit tests from installing the opencv-python/mozperftest-tools module during unit tests.
Differential Revision: https://phabricator.services.mozilla.com/D195901
This patch updates mozperftest-tools to 0.2.9, and removes the opencv-python 4.5 install since it interferes with the new opencv-python package that has a lower minimum version for python (version 3.7).
Differential Revision: https://phabricator.services.mozilla.com/D195774
This patch fixes an issue where only `perftest.ini` manifests were being parsed for mozperftest tests in perfdocs. It's temporary as we will move all ini manifests to toml in the future. At the same time, this patch adds missing documentation for some new mozperftest tests that use a toml manifest to define the tests.
Differential Revision: https://phabricator.services.mozilla.com/D194251
These are a first-cut of service worker performance tests, just running as
vanilla mochitests for now. Bug 1832059 tracks the integration work necessary.
Differential Revision: https://phabricator.services.mozilla.com/D177427
These are a first-cut of service worker performance tests, just running as
vanilla mochitests for now. Bug 1832059 tracks the integration work necessary.
Differential Revision: https://phabricator.services.mozilla.com/D177427
This patch sets up the paths for DMG extraction for Mochitest mozperftest tests. It also adds some additional artifacts for the mac tests to pull and extract.
Differential Revision: https://phabricator.services.mozilla.com/D192071
This patch adds some documentation on how to write Mochitest tests that are compatible with mozperftest. It also provides some additional information about running the tests, setting them up, and some common issues that people can hit.
Differential Revision: https://phabricator.services.mozilla.com/D192019
This patch adds some code to handle when the virtualenv lib path changes between the different platforms. In this case, the change makes it possible to use mochitest-mozperftest on winows machines in CI. It also adds some additional artifacts that the windows tests need to pull.
Differential Revision: https://phabricator.services.mozilla.com/D191741
This patch adds a new layer that allows developers to run mochitest tests as performance tests directly. It provides the ability to run them in CI on linux, and locally through `./mach test` using the FunctionalTestRunner. Some additional try mapping is also added for future tests. The remote run (CI) method uses the mochitest run_test_harness method directly, along with the mochitest argument parser.
Differential Revision: https://phabricator.services.mozilla.com/D190994
This patch adds a FunctionalTestRunner class that can be used by all the layers to run `./mach test` which lets us easily run all functional tests locally. It will be used for running mochitest tests, and in the future, it may replace parts of the xpcshell layer.
The runner can only be used locally, and only needs a test path provided. It will then run all the manifests that it finds that test in (but only running that single test). At the same time, it'll parse the logs produced by `./mach test` with FunctionalTestProcessor to find all the metrics that were output. However, it does not parse them into a JSON format, that's left for the layers to handle.
Differential Revision: https://phabricator.services.mozilla.com/D190993
These methods are added as we need to install a requirement file for mochitest (marionette), and we also need to import a class directly from a file. The existing methods we have for importing, and installation don't work for these cases.
Differential Revision: https://phabricator.services.mozilla.com/D190992
This patch adds the ability to parse HTML files (mochitest test files) to ScriptInfo. This makes it possible to parse the `perfMetadata` from the mochitest tests which will provide mozperftest with default options, and metadata for our performance docs.
Differential Revision: https://phabricator.services.mozilla.com/D190990
A week ago we got a permafail when requesting to know the location queue for certain data centres for webpagetest
I investigated and found the request was being blocked by cloudflare for some security reasons
Modifying the request URL and modifying headers resolved the issue
Differential Revision: https://phabricator.services.mozilla.com/D187497
Since various dependencies were moved out of the `mach` site, activating just it is no longer sufficient. Activating the `common` site solves this problem, since it has everything that was moved out of the `mach` site.
Differential Revision: https://phabricator.services.mozilla.com/D184662
The artifacts where the fenix and focus apks has changed locations and artifact names, this patch is altering the mozperftest fenix and focus startup test to use the new location and file names
Differential Revision: https://phabricator.services.mozilla.com/D183601
Currently the new platform and task names are getting stuck as the first test being analyzed. This patch fixes it by changing how those two values are setup.
Differential Revision: https://phabricator.services.mozilla.com/D175246
This patch adds a new tool that runs a comparison between two, or more revisions to detect performance changes. Some changes are made to accommodate this new tool alongside the side-by-side tool. The tests for the detector coding are found in another patch in the series. A mozperftest-tools update to 0.2.5 is required for this change.
There is also a CI task that is added in this patch. It's setup in the mach try perf patch in this series, which also has more information.
Differential Revision: https://phabricator.services.mozilla.com/D172282
This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.
The `./mach try perf` command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the `./mach try perf` command to select the appropriate tests.
Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.
Differential Revision: https://phabricator.services.mozilla.com/D172435
This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.
The `./mach try perf` command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the `./mach try perf` command to select the appropriate tests.
Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.
Differential Revision: https://phabricator.services.mozilla.com/D172435