Commit Graph
322 Commits
Author SHA1 Message Date
Mike Hommey 32831b48cc Bug 1897317 - Rename MOZ_ANDROID_HISTORY to MOZ_GECKOVIEW_HISTORY. r=firefox-build-system-reviewers,geckoview-reviewers,nalexander,owlish
Differential Revision: https://phabricator.services.mozilla.com/D210743
2024-05-17 00:29:35 +00:00
KS f8d2172321 Bug 1887998 - Have mozperftest and raptor use the same browsertime. r=perftest-reviewers,sparky
Currently, the browsertime used by mozperftest is outdated and
maintained separately from the browsertime used in raptor tests.

This patch upgrades the browsertime used in mozperftest by now sharing
the same browsertime that is used with raptors (in `src/tools/browsertime`).

Going forward we now only have to maintain one browsertime version. This
introduces the potential of more infra changes (because previously when
upgrading raptor's browsertime we only checked raptor tests, but now we
will have to verify mozperftest) but this should be worth it to stay up
to date.

Differential Revision: https://phabricator.services.mozilla.com/D208944
2024-05-14 14:59:05 +00:00
andrej 5e2dd9d283 Bug 1884930 - Use per-commit fenix and geckoview builds instead of downloading daily build. r=perftest-reviewers,taskgraph-reviewers,sparky,bhearsum
As a result of the mono-repo change we have a very easy way to run the startup tests on each commit vs on a nightly build.
This change allows us to push a try job, build the fenix/focus/geckoview_example apk and test the startup impact!

Differential Revision: https://phabricator.services.mozilla.com/D208240
2024-05-13 16:50:03 +00:00
Iulian Moraru f4afd9c509 Backed out changeset d4ef5f3b9858 (bug 1884930) for causing Bug 1896049. CLOSED TREE 2024-05-10 17:38:49 +03:00
andrej 54e1b45e94 Bug 1884930 - Use per-commit fenix and geckoview builds instead of downloading daily build. r=perftest-reviewers,taskgraph-reviewers,sparky,bhearsum
As a result of the mono-repo change we have a very easy way to run the startup tests on each commit vs on a nightly build.
This change allows us to push a try job, build the fenix/focus/geckoview_example apk and test the startup impact!

Differential Revision: https://phabricator.services.mozilla.com/D208240
2024-05-09 15:05:05 +00:00
Henrik Skupin 7c80d55b26 Bug 1892044 - [mozperftest] Remove "--no-deps" argument for installing Python requirements file. r=sparky
Depends on D207902

Differential Revision: https://phabricator.services.mozilla.com/D207903
2024-04-19 01:53:11 +00:00
Andrew Halberstadt a680bcd3e1 Bug 1868440 - [ci] Upgrade taskcluster-taskgraph to v7.4.0, r=perftest-reviewers,geckoview-reviewers,ci-and-tooling,devtools-reviewers,taskgraph-reviewers,releng-reviewers,mach-reviewers,omc-reviewers,jmaher,hneiva,aminomancer,jari,dom-storage-reviewers,afinder,asuth,bhearsum,owlish
Differential Revision: https://phabricator.services.mozilla.com/D206260
2024-04-18 20:39:09 +00:00
Stanca Serban 3dbb534055 Backed out 2 changesets (bug 1868440) for gecko decision task broken. CLOSED TREE
Backed out changeset 4b22948d849a (bug 1868440)
Backed out changeset 837bdcb3694a (bug 1868440)
2024-04-18 21:25:43 +03:00
Andrew Halberstadt ef4f1e168b Bug 1868440 - [ci] Upgrade taskcluster-taskgraph to v7.4.0, r=perftest-reviewers,geckoview-reviewers,ci-and-tooling,devtools-reviewers,taskgraph-reviewers,releng-reviewers,mach-reviewers,omc-reviewers,jmaher,hneiva,aminomancer,jari,dom-storage-reviewers,afinder,asuth,bhearsum,owlish
Differential Revision: https://phabricator.services.mozilla.com/D206260
2024-04-18 17:50:50 +00:00
andrej c636c101a3 Bug 1888142 - Webpagetest Permafail due to new way of accepting API keys. r=perftest-reviewers,kshampur
Webpagetest has modified the way that they accept keys, they used to accept it through a URL parameter, but now they require it to be in the header and with a different name

Differential Revision: https://phabricator.services.mozilla.com/D207159
2024-04-10 18:44:46 +00:00
andrej 13b8db92e1 Bug 1836330 - Enable android chrome startup testing for cold_main_first_frame. r=perftest-reviewers,sparky
In this commit we are adding the ability to run the cold_main_first_frame startup test on android google chrome

Differential Revision: https://phabricator.services.mozilla.com/D202626
2024-03-26 12:51:54 +00:00
andrej 938a7ccc52 Bug 1885080 - Modify links used to download browser apks for android_startup script. r=perftest-reviewers,afinder
As the monorepo migration is occurring we want to access the apks in the new place they are located in post-migration

Differential Revision: https://phabricator.services.mozilla.com/D205014
2024-03-19 15:35:24 +00:00
ahochheiden 0715082d45 Bug 1877344 - Replace all usages of distutils.spawn.find_executable with shutil.which within mach r=firefox-build-system-reviewers,glandium
Differential Revision: https://phabricator.services.mozilla.com/D199949
2024-02-08 21:07:26 +00:00
Greg Mierzwinski 07315862c7 Bug 1877180 - Pass the app name to mozproxy during recording. r=perftest-reviewers,kshampur,afinder DONTBUILD
This patch fixes a couple bugs with recording. The first is that the app wasn't being passed to mozproxy, and this was preventing the creation of the mitmproxy certificate for the app being used to record. The second is that the --proxy-deterministic flag isn't defined within the proxy layer even though it's used by it from the hooks.

Differential Revision: https://phabricator.services.mozilla.com/D199891
2024-02-02 13:09:18 +00:00
andrej 984dfe840e Bug 1874774 - Remove references to moto g5 and pixel2 devices in taskcluster. r=perftest-reviewers,jmaher,sparky
As the motorola G5 and the Pixel 2 have been removed from CI we are also going to remove references to them in our documentation and taskcluster code.

Differential Revision: https://phabricator.services.mozilla.com/D199294
2024-01-31 02:42:08 +00:00
andrej ea3318d60e Bug 1870500 - Resolving the WPT intermittent failure. r=perftest-reviewers,kshampur
The issue we were encountering here was a missing metric for canva.com intermittently occuring
That ocured because of the location of that metric(first contentful paint) in the returned json moved locations for just canva.com, and so canva.com was removed whilst the problem debugged. Afterwards it was discovered that this issue was covering up another intermittent about json decoding error which was in turn resolved by removing code which removed "www." from links used with python requests.

Differential Revision: https://phabricator.services.mozilla.com/D199238
2024-01-24 02:36:24 +00:00
Alex Finder c6c5fed905 Bug 1870051 Update mozperftest-tools version to 0.3.2 r=perftest-reviewers,sparky
Differential Revision: https://phabricator.services.mozilla.com/D196428
2024-01-08 16:25:43 +00:00
Greg Mierzwinski 14718089e9 Bug 1870864 - Fix mozperftest writing docs to mention mochitest. r=aglavic,perftest-reviewers DONTBUILD
This patch fixes the mozperftest docs to mention mochitest in the writing section, as well as fixes the `info` calls so that they mention the JSON.stringify requirement.

Differential Revision: https://phabricator.services.mozilla.com/D196853
2024-01-08 15:15:45 +00:00
Joel Maher 6d618a2524 Bug 1869871 - convert perftest.ini -> .toml. r=sparky,perftest-reviewers,necko-reviewers,jesup
Differential Revision: https://phabricator.services.mozilla.com/D196334
2023-12-14 19:41:14 +00:00
Joel Maher 6ef3fa1746 Bug 1869437 - support .toml for mozperftest when using xpcshell. r=sparky
Differential Revision: https://phabricator.services.mozilla.com/D196190
2023-12-12 17:34:41 +00:00
Greg Mierzwinski d00f013bfc Bug 1868937 - Prevent install of opencv during mozperftest unit tests.r=aglavic
This patch prevents the mozperftest unit tests from installing the opencv-python/mozperftest-tools module during unit tests.

Differential Revision: https://phabricator.services.mozilla.com/D195901
2023-12-11 14:56:43 +00:00
Greg Mierzwinski eddbfd58d7 Bug 1867956 - Remove opencv-python 4.5 install, and update mpt to 0.2.9. r=perftest-reviewers,AlexandruIonescu DONTBUILD
This patch updates mozperftest-tools to 0.2.9, and removes the opencv-python 4.5 install since it interferes with the new opencv-python package that has a lower minimum version for python (version 3.7).

Differential Revision: https://phabricator.services.mozilla.com/D195774
2023-12-07 14:17:54 +00:00
Greg Mierzwinski 2e4628dcf9 Bug 1865869 - Allow toml files to be used in perfdocs generation. r=jmarshall,perftest-reviewers,afinder DONTBUILD
This patch fixes an issue where only `perftest.ini` manifests were being parsed for mozperftest tests in perfdocs. It's temporary as we will move all ini manifests to toml in the future. At the same time, this patch adds missing documentation for some new mozperftest tests that use a toml manifest to define the tests.

Differential Revision: https://phabricator.services.mozilla.com/D194251
2023-12-06 13:01:24 +00:00
Joshua Marshall d65088640e Bug 1299271 - Service worker performance test r=asuth,sparky
These are a first-cut of service worker performance tests, just running as
vanilla mochitests for now.  Bug 1832059 tracks the integration work necessary.

Differential Revision: https://phabricator.services.mozilla.com/D177427
2023-11-28 16:33:38 +00:00
Sandor Molnar ca82695b2b Backed out changeset 6a14206eb5a9 (bug 1299271) for causing python taskgraph-tests failures at taskcluster/gecko_taskgraph/test/test_util_chunking.py CLOSED TREE 2023-11-22 22:50:30 +02:00
Joshua Marshall bb47d44e7b Bug 1299271 - Service worker performance test r=asuth,sparky
These are a first-cut of service worker performance tests, just running as
vanilla mochitests for now.  Bug 1832059 tracks the integration work necessary.

Differential Revision: https://phabricator.services.mozilla.com/D177427
2023-11-22 13:16:11 +00:00
Greg Mierzwinski a88fcbc65b Bug 1856681 - Use proper venv for perftest-tools, and accept output argument. r=perftest-reviewers,AlexandruIonescu
Differential Revision: https://phabricator.services.mozilla.com/D189962
2023-11-21 16:59:06 +00:00
Greg Mierzwinski 76fd189c7f Bug 1860189 - Setup path for DMG extraction for Mac Mochitest mozperftest. r=perftest-reviewers,kshampur
This patch sets up the paths for DMG extraction for Mochitest mozperftest tests. It also adds some additional artifacts for the mac tests to pull and extract.

Differential Revision: https://phabricator.services.mozilla.com/D192071
2023-11-07 14:37:25 +00:00
Greg Mierzwinski 0820b59751 Bug 1861506 - Add docs on how to write mozperftest Mochitest tests. r=afinder,perftest-reviewers DONTBUILD
This patch adds some documentation on how to write Mochitest tests that are compatible with mozperftest. It also provides some additional information about running the tests, setting them up, and some common issues that people can hit.

Differential Revision: https://phabricator.services.mozilla.com/D192019
2023-10-27 14:39:48 +00:00
Greg Mierzwinski b3789c16b0 Bug 1860189 - Handle virtualenv paths correctly when installing requirements. r=aglavic,perftest-reviewers
This patch adds some code to handle when the virtualenv lib path changes between the different platforms. In this case, the change makes it possible to use mochitest-mozperftest on winows machines in CI. It also adds some additional artifacts that the windows tests need to pull.

Differential Revision: https://phabricator.services.mozilla.com/D191741
2023-10-26 17:57:11 +00:00
Greg Mierzwinski 1340e24dcf Bug 1832059 - Add a layer to run mochitest performance tests. r=perftest-reviewers,kshampur
This patch adds a new layer that allows developers to run mochitest tests as performance tests directly. It provides the ability to run them in CI on linux, and locally through `./mach test` using the FunctionalTestRunner. Some additional try mapping is also added for future tests. The remote run (CI) method uses the mochitest run_test_harness method directly, along with the mochitest argument parser.

Differential Revision: https://phabricator.services.mozilla.com/D190994
2023-10-24 13:05:50 +00:00
Greg Mierzwinski e92dd37363 Bug 1832059 - Add a test runner for functional tests to mozperftest. r=perftest-reviewers,kshampur
This patch adds a FunctionalTestRunner class that can be used by all the layers to run `./mach test` which lets us easily run all functional tests locally. It will be used for running mochitest tests, and in the future, it may replace parts of the xpcshell layer.

The runner can only be used locally, and only needs a test path provided. It will then run all the manifests that it finds that test in (but only running that single test). At the same time, it'll parse the logs produced by `./mach test` with FunctionalTestProcessor to find all the metrics that were output. However, it does not parse them into a JSON format, that's left for the layers to handle.

Differential Revision: https://phabricator.services.mozilla.com/D190993
2023-10-24 13:05:49 +00:00
Greg Mierzwinski 7e7d6d3f67 Bug 1832059 - Add new methods for importing, and installing to mozperftest. r=perftest-reviewers,aglavic
These methods are added as we need to install a requirement file for mochitest (marionette), and we also need to import a class directly from a file. The existing methods we have for importing, and installation don't work for these cases.

Differential Revision: https://phabricator.services.mozilla.com/D190992
2023-10-24 13:05:49 +00:00
Greg Mierzwinski fe48e26fea Bug 1832059 - Modify ScriptInfo to handle mochitest html files. r=perftest-reviewers,AlexandruIonescu
This patch adds the ability to parse HTML files (mochitest test files) to ScriptInfo. This makes it possible to parse the `perfMetadata` from the mochitest tests which will provide mozperftest with default options, and metadata for our performance docs.

Differential Revision: https://phabricator.services.mozilla.com/D190990
2023-10-24 13:05:48 +00:00
Alex Finder 0816602e61 Bug 1843139 Update mozperftest-tools to 0.2.8 r=perftest-reviewers,AlexandruIonescu
Differential Revision: https://phabricator.services.mozilla.com/D189915
2023-10-23 11:04:46 +00:00
andrej d7a37374bb Bug 1850665 - Alter location URL and request headers to not be blocked when requesting location queue. r=perftest-reviewers,kshampur
A week ago we got a permafail when requesting to know the location queue for certain data centres for webpagetest
I investigated and found the request was being blocked by cloudflare for some security reasons
Modifying the request URL and modifying headers resolved the issue

Differential Revision: https://phabricator.services.mozilla.com/D187497
2023-09-07 16:12:48 +00:00
Logan Rosen 45be05a920 Bug 1786490 - reformat the tree using black 23.3.0 r=sylvestre,webdriver-reviewers,taskgraph-reviewers,whimboo,jmaher,ahal,perftest-reviewers,geckoview-reviewers,afinder,m_kato
# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D186092
2023-09-06 16:14:30 +00:00
Logan Rosen 673c4dae7f Bug 1786490 - remove references to vendored pyparsing r=sylvestre,mach-reviewers,ahal,ahochheiden
Not necessary because pyparsing is no longer a dependency of packaging, so it is no longer vendored.

Differential Revision: https://phabricator.services.mozilla.com/D186411
2023-09-06 16:14:30 +00:00
ahochheiden 3ac3144e34 Bug 1845272 - Activate the common site virtualenv when calling the mozperftest/runner.py script directly (outside of a mach command context) r=ahal
Since various dependencies were moved out of the `mach` site, activating just it is no longer sufficient. Activating the `common` site solves this problem, since it has everything that was moved out of the `mach` site.

Differential Revision: https://phabricator.services.mozilla.com/D184662
2023-07-27 16:39:35 +00:00
andrej 773211dd9a Bug 1843222 - Fenix and Focus download perma fail, link to downloading fenix and focus apk has changed. r=perftest-reviewers,sparky
The artifacts where the fenix and focus apks has changed locations and artifact names, this patch is altering the mozperftest fenix and focus startup test to use the new location and file names

Differential Revision: https://phabricator.services.mozilla.com/D183601
2023-07-17 15:56:59 +00:00
andrej ed02242adb Bug 1816169 - Remove perftest_android_main.js and perftest_android_view.js from nightly cron. r=perftest-reviewers,sparky
Differential Revision: https://phabricator.services.mozilla.com/D180724
2023-06-20 13:17:01 +00:00
Greg Mierzwinski 47c733eb0b Bug 1827609 - Set new platform/test names correctly when using MWU to detect changes. r=perftest-reviewers,kshampur
Currently the new platform and task names are getting stuck as the first test being analyzed. This patch fixes it by changing how those two values are setup.

Differential Revision: https://phabricator.services.mozilla.com/D175246
2023-04-18 17:05:20 +00:00
andrej 4f213b4fe1 Bug 1827256 - Run Nightly start up benchmarks on geckoview_example. r=perftest-reviewers,sparky
Differential Revision: https://phabricator.services.mozilla.com/D175386
2023-04-18 13:23:49 +00:00
andrej 567202479f Bug 1812092 - Update startup tests to use a fixed current date. r=perftest-reviewers,AlexandruIonescu
Differential Revision: https://phabricator.services.mozilla.com/D173247
2023-04-11 19:11:07 +00:00
Paul Adenot f3638df0fd Bug 1825554 - Use monotonic clock to report durations in mach. r=firefox-build-system-reviewers,sergesanspaille
Differential Revision: https://phabricator.services.mozilla.com/D174118
2023-04-04 11:13:30 +00:00
Greg Mierzwinski b7c04ecbac Bug 1821708 - Add tests for the change-detector.r=perftest-reviewers,afinder
This patch adds tests for the new change-detector tool.

Differential Revision: https://phabricator.services.mozilla.com/D172283
2023-03-28 11:16:52 +00:00
Greg Mierzwinski abd72ed08f Bug 1821708 - Add performance change detector to mach perftest-tools. r=perftest-reviewers,kshampur,AlexandruIonescu
This patch adds a new tool that runs a comparison between two, or more revisions to detect performance changes. Some changes are made to accommodate this new tool alongside the side-by-side tool. The tests for the detector coding are found in another patch in the series. A mozperftest-tools update to 0.2.5 is required for this change.

There is also a CI task that is added in this patch. It's setup in the mach try perf patch in this series, which also has more information.

Differential Revision: https://phabricator.services.mozilla.com/D172282
2023-03-28 11:16:51 +00:00
Greg Mierzwinski d52f3c8277 Bug 1613455 - Allow custom APK uploads for Geckoview/Fenix perftests. r=perftest-reviewers,afinder
This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.

The `./mach try perf` command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the `./mach try perf` command to select the appropriate tests.

Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.

Differential Revision: https://phabricator.services.mozilla.com/D172435
2023-03-23 13:43:45 +00:00
Iulian Moraru 61f3d1f543 Backed out changeset faa1c03828fe (bug 1613455) for causing btime failures. CLOSED TREE 2023-03-22 19:10:02 +02:00
Greg Mierzwinski 61a4365117 Bug 1613455 - Allow custom APK uploads for Geckoview/Fenix perftests. r=perftest-reviewers,afinder
This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.

The `./mach try perf` command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the `./mach try perf` command to select the appropriate tests.

Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.

Differential Revision: https://phabricator.services.mozilla.com/D172435
2023-03-22 12:30:17 +00:00