This patch fixes a couple bugs with recording. The first is that the app wasn't being passed to mozproxy, and this was preventing the creation of the mitmproxy certificate for the app being used to record. The second is that the --proxy-deterministic flag isn't defined within the proxy layer even though it's used by it from the hooks.
Differential Revision: https://phabricator.services.mozilla.com/D199891
As the motorola G5 and the Pixel 2 have been removed from CI we are also going to remove references to them in our documentation and taskcluster code.
Differential Revision: https://phabricator.services.mozilla.com/D199294
The issue we were encountering here was a missing metric for canva.com intermittently occuring
That ocured because of the location of that metric(first contentful paint) in the returned json moved locations for just canva.com, and so canva.com was removed whilst the problem debugged. Afterwards it was discovered that this issue was covering up another intermittent about json decoding error which was in turn resolved by removing code which removed "www." from links used with python requests.
Differential Revision: https://phabricator.services.mozilla.com/D199238
This patch fixes the mozperftest docs to mention mochitest in the writing section, as well as fixes the `info` calls so that they mention the JSON.stringify requirement.
Differential Revision: https://phabricator.services.mozilla.com/D196853
This patch prevents the mozperftest unit tests from installing the opencv-python/mozperftest-tools module during unit tests.
Differential Revision: https://phabricator.services.mozilla.com/D195901
This patch updates mozperftest-tools to 0.2.9, and removes the opencv-python 4.5 install since it interferes with the new opencv-python package that has a lower minimum version for python (version 3.7).
Differential Revision: https://phabricator.services.mozilla.com/D195774
This patch fixes an issue where only `perftest.ini` manifests were being parsed for mozperftest tests in perfdocs. It's temporary as we will move all ini manifests to toml in the future. At the same time, this patch adds missing documentation for some new mozperftest tests that use a toml manifest to define the tests.
Differential Revision: https://phabricator.services.mozilla.com/D194251
These are a first-cut of service worker performance tests, just running as
vanilla mochitests for now. Bug 1832059 tracks the integration work necessary.
Differential Revision: https://phabricator.services.mozilla.com/D177427
These are a first-cut of service worker performance tests, just running as
vanilla mochitests for now. Bug 1832059 tracks the integration work necessary.
Differential Revision: https://phabricator.services.mozilla.com/D177427
This patch sets up the paths for DMG extraction for Mochitest mozperftest tests. It also adds some additional artifacts for the mac tests to pull and extract.
Differential Revision: https://phabricator.services.mozilla.com/D192071
This patch adds some documentation on how to write Mochitest tests that are compatible with mozperftest. It also provides some additional information about running the tests, setting them up, and some common issues that people can hit.
Differential Revision: https://phabricator.services.mozilla.com/D192019
This patch adds some code to handle when the virtualenv lib path changes between the different platforms. In this case, the change makes it possible to use mochitest-mozperftest on winows machines in CI. It also adds some additional artifacts that the windows tests need to pull.
Differential Revision: https://phabricator.services.mozilla.com/D191741
This patch adds a new layer that allows developers to run mochitest tests as performance tests directly. It provides the ability to run them in CI on linux, and locally through `./mach test` using the FunctionalTestRunner. Some additional try mapping is also added for future tests. The remote run (CI) method uses the mochitest run_test_harness method directly, along with the mochitest argument parser.
Differential Revision: https://phabricator.services.mozilla.com/D190994
This patch adds a FunctionalTestRunner class that can be used by all the layers to run `./mach test` which lets us easily run all functional tests locally. It will be used for running mochitest tests, and in the future, it may replace parts of the xpcshell layer.
The runner can only be used locally, and only needs a test path provided. It will then run all the manifests that it finds that test in (but only running that single test). At the same time, it'll parse the logs produced by `./mach test` with FunctionalTestProcessor to find all the metrics that were output. However, it does not parse them into a JSON format, that's left for the layers to handle.
Differential Revision: https://phabricator.services.mozilla.com/D190993
These methods are added as we need to install a requirement file for mochitest (marionette), and we also need to import a class directly from a file. The existing methods we have for importing, and installation don't work for these cases.
Differential Revision: https://phabricator.services.mozilla.com/D190992
This patch adds the ability to parse HTML files (mochitest test files) to ScriptInfo. This makes it possible to parse the `perfMetadata` from the mochitest tests which will provide mozperftest with default options, and metadata for our performance docs.
Differential Revision: https://phabricator.services.mozilla.com/D190990
A week ago we got a permafail when requesting to know the location queue for certain data centres for webpagetest
I investigated and found the request was being blocked by cloudflare for some security reasons
Modifying the request URL and modifying headers resolved the issue
Differential Revision: https://phabricator.services.mozilla.com/D187497
Since various dependencies were moved out of the `mach` site, activating just it is no longer sufficient. Activating the `common` site solves this problem, since it has everything that was moved out of the `mach` site.
Differential Revision: https://phabricator.services.mozilla.com/D184662
The artifacts where the fenix and focus apks has changed locations and artifact names, this patch is altering the mozperftest fenix and focus startup test to use the new location and file names
Differential Revision: https://phabricator.services.mozilla.com/D183601
Currently the new platform and task names are getting stuck as the first test being analyzed. This patch fixes it by changing how those two values are setup.
Differential Revision: https://phabricator.services.mozilla.com/D175246
This patch adds a new tool that runs a comparison between two, or more revisions to detect performance changes. Some changes are made to accommodate this new tool alongside the side-by-side tool. The tests for the detector coding are found in another patch in the series. A mozperftest-tools update to 0.2.5 is required for this change.
There is also a CI task that is added in this patch. It's setup in the mach try perf patch in this series, which also has more information.
Differential Revision: https://phabricator.services.mozilla.com/D172282
This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.
The `./mach try perf` command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the `./mach try perf` command to select the appropriate tests.
Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.
Differential Revision: https://phabricator.services.mozilla.com/D172435
This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.
The `./mach try perf` command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the `./mach try perf` command to select the appropriate tests.
Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.
Differential Revision: https://phabricator.services.mozilla.com/D172435
This patch adds the ability to run manual logins for our websites since it can be simpler, and quicker in some cases. At the same time, a bug with the options handling is fixed.
Differential Revision: https://phabricator.services.mozilla.com/D164590
This results in some changes from our current `isort` configuration. I'm
unclear if it's because ruff isn't at 100% parity with isort, they choose
different defaults or if I missed some configuration.
Either way, the changes all look reasonable to me (see child commit), so I'm
inclined to just accept the new import format it imposes.
Differential Revision: https://phabricator.services.mozilla.com/D172348
This results in some changes from our current `isort` configuration. I'm
unclear if it's because ruff isn't at 100% parity with isort, they choose
different defaults or if I missed some configuration.
Either way, the changes all look reasonable to me (see child commit), so I'm
inclined to just accept the new import format it imposes.
Differential Revision: https://phabricator.services.mozilla.com/D172348
This patch fixes an issue in our perftestetl.py which was causing our coverage percent to drop "artificially". This is happening because there's a bit of code that's unreachable because of an exception that would always be triggered before it. This patch removes the unreachable code, and replaces it with an exception.
Differential Revision: https://phabricator.services.mozilla.com/D172281
This patch updates the existing tests so that they're inline with the new default of providing no summary value at the suite level. A new test is also added to test out custom transforms.
Differential Revision: https://phabricator.services.mozilla.com/D171613
This patch removes the default summary value for perfherder suites. Afterwards, no summary value will be produced unless a new Transformer is provided, or the existing ones are modified.
The patch goes further and creates a path through which we can let users modify the suites, and subtest in any way they want. This is done by adding two new methods that can be added to the transformer being used on the data. Some changes also needed to be made to the MetricsStorage so that we keep track of the transformer. They can be specified on a per-subtest basis some day, but right now it applies a single transformer to all subtests of a given suite/data-type.
Differential Revision: https://phabricator.services.mozilla.com/D171612
