Commit Graph
449 Commits
Author SHA1 Message Date
KS 391e8f49c7 Bug 1840660 - Add documentation on running Safari-tp locally. r=perftest-reviewers,aglavic
This patch updates our documents to include a section with instructions
on how a user can run a browsertime or raptor-browsertime test locally
with safari tech preview

Differential Revision: https://phabricator.services.mozilla.com/D209878
2024-05-14 14:48:20 +00:00
KS 6cbb2d6a1d Bug 1840660 - Run performance tests against safari technology preview. r=perftest-reviewers,sparky
This patch adds the ability to run Raptor performance tests
(specifically just speedometer 3 for now) on Safari Technology Preview.

Differential Revision: https://phabricator.services.mozilla.com/D209611
2024-05-14 14:48:20 +00:00
Greg Mierzwinski 790461e195 Bug 1894367 - Update perfdocs with talos changes. r=perftest-reviewers,kshampur
This patch updates perfdocs with talos doc changes, and makes a modification to the perfdocs-verify task to trigger it on reviewbot again.

Differential Revision: https://phabricator.services.mozilla.com/D209228
2024-05-03 14:25:22 +00:00
Sandor Molnar b2bb9fd8f8 Backed out changeset e22820789eaa (bug 1894367) because conflicts with backout of bug 1885361. 2024-05-06 06:15:29 +03:00
Greg Mierzwinski 35ec83451b Bug 1894367 - Update perfdocs with talos changes. r=perftest-reviewers,kshampur DONTBUILD
This patch updates perfdocs with talos doc changes, and makes a modification to the perfdocs-verify task to trigger it on reviewbot again.

Differential Revision: https://phabricator.services.mozilla.com/D209228
2024-05-03 14:25:22 +00:00
Greg Mierzwinski 394048a744 Bug 1894367 - Update perfdocs with profiler features changes. r=kshampur,perftest-reviewers DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D209133
2024-05-01 18:55:45 +00:00
Greg Mierzwinski 8de37416cd Bug 1893972 - Add info about 'Use Replicates' button to mach-try-perf docs. r=afinder,perftest-reviewers DONTBUILD
This patch adds a couple sentences to our standard-workflow docs for mach-try-perf to explain the use replicates button.

Differential Revision: https://phabricator.services.mozilla.com/D208877
2024-04-30 15:55:39 +00:00
KS 20d1688553 Bug 1892470 - Fix x64 chromedriver fetch for windows raptor tests. r=perftest-reviewers,taskgraph-reviewers,sparky,bhearsum
Grab the correct folder after extracting the x64 chromedrivers.
Previously we were always using the 32 bit chromedrivers in CI and only
recently added the 64 bit chromedrivers for win x64 platforms.

Additionally, since we don't make use of the 32 bit windows platform
anymore, it is removed for the chrome platform.

Differential Revision: https://phabricator.services.mozilla.com/D208001
2024-04-23 13:21:52 +00:00
Andrew Halberstadt a680bcd3e1 Bug 1868440 - [ci] Upgrade taskcluster-taskgraph to v7.4.0, r=perftest-reviewers,geckoview-reviewers,ci-and-tooling,devtools-reviewers,taskgraph-reviewers,releng-reviewers,mach-reviewers,omc-reviewers,jmaher,hneiva,aminomancer,jari,dom-storage-reviewers,afinder,asuth,bhearsum,owlish
Differential Revision: https://phabricator.services.mozilla.com/D206260
2024-04-18 20:39:09 +00:00
Stanca Serban 3dbb534055 Backed out 2 changesets (bug 1868440) for gecko decision task broken. CLOSED TREE
Backed out changeset 4b22948d849a (bug 1868440)
Backed out changeset 837bdcb3694a (bug 1868440)
2024-04-18 21:25:43 +03:00
Andrew Halberstadt ef4f1e168b Bug 1868440 - [ci] Upgrade taskcluster-taskgraph to v7.4.0, r=perftest-reviewers,geckoview-reviewers,ci-and-tooling,devtools-reviewers,taskgraph-reviewers,releng-reviewers,mach-reviewers,omc-reviewers,jmaher,hneiva,aminomancer,jari,dom-storage-reviewers,afinder,asuth,bhearsum,owlish
Differential Revision: https://phabricator.services.mozilla.com/D206260
2024-04-18 17:50:50 +00:00
KS 19a7f668b2 Bug 1890657 - Re-record a couple non-login sites for OSX intermittent (expedia, microsoft). r=perftest-reviewers,afinder
This patch re-records expedia and microsoft. Hopefully this will help a
bit with some intermittents (Bug 1774954)

Differential Revision: https://phabricator.services.mozilla.com/D207195
2024-04-17 15:40:59 +00:00
Greg Mierzwinski df2d4e7105 Bug 1890318 - Add missing metrics to Raptor documentation. r=perftest-reviewers,kshampur DONTBUILD
This patch adds the final missing metrics that alert to the Raptor documentation.

Differential Revision: https://phabricator.services.mozilla.com/D206952
2024-04-17 12:34:30 +00:00
Greg Mierzwinski 3ee37077ea Bug 1890318 - Add dynamic metric documentation to perfdocs. r=perftest-reviewers,aglavic DONTBUILD
This patch adds the ability to document metrics dynamically. The metrics are documented in the config YAML file for each framework, and then defined in a metrics.rst file. Two things are expected to be found in the framework docs now if they want to use dynamic metric documentation:
* A metrics.rst file that contains a `{metrics_documentation}` entry somewhere to insert the built documentation.
* A `{metrics_rst_name}` entry somewhere in the top-level/index.rst file so that a link can be produced to the metrics documentation.

The metrics documentation is specific to each framework. See the schema for the metrics for what is expected in this field. At the same time, the dynamic documentation for Raptor is generated here. Each metric in Raptor contains a list of all the tests that use (alert on) the specific metric. The tests link back to the test definition, and the metrics defined in the tests also link to the metric definitions.

Differential Revision: https://phabricator.services.mozilla.com/D206950
2024-04-17 12:34:29 +00:00
KS 399e300dc1 Bug 1890059 - Update raptor config to handle tp7. r=perftest-reviewers,sparky
This patch enables the ability to run tp7 tests in raptor-browsertime.
At the moment these tests will not run on CI, but are available to run
locally. This is merely preliminary work on desktop that will help when
shifting focus to the android platform.

Differential Revision: https://phabricator.services.mozilla.com/D206826
2024-04-10 18:11:10 +00:00
KS 99c4aee37b Bug 1889241 - re-record some sites for OSX failures. r=perftest-reviewers,sparky
This bug re-records a subset of some sites that are potentially causing
an intermittent issue (Bug 1774954)
Here, only `amazon`, `imgur`, and `instagram` are re-recorded. Other
sites may be re-recorded in a follow up bug.

Differential Revision: https://phabricator.services.mozilla.com/D206572
2024-04-09 16:39:03 +00:00
Greg Mierzwinski 269ba73e5a Bug 1890496 - Update pdfpaint description to fix doc generate warnings. r=kshampur,perftest-reviewers DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D207015
2024-04-09 13:29:17 +00:00
Greg Mierzwinski 3185185c83 Bug 1855674 - Run pdfpaint talos test in CI. r=kshampur,perftest-reviewers,taskgraph-reviewers,ahal
This patch adds a set of tasks for the talos pdfpaint test in CI. It'll run with, and without the swr variant on linux, windows, and mac. The number of chunks are defined manually in a taskcluster transform where the test is chunked.

Differential Revision: https://phabricator.services.mozilla.com/D205825
2024-04-08 11:47:05 +00:00
Greg Mierzwinski 4f87579351 Bug 1855674 - Modify pdfpaint test to run more PDFs. r=aglavic,perftest-reviewers
This patch modifies the pdfpaint test to run more pdfs that are found in the Mozilla pdf.js repository. The pdfpaint test is also moved to it's own suite due to the number of PDFs now being tested. These PDFs are pulled in locally from a toolchain task called talos-pdfs. The *ignore files are modified since the pdfpaint folder now contains a symbolic link to the local PDFs that should not be commit in-tree.

To handle running the large number of PDFs, chunking is added to the test with the chunk size being 100 PDFs. Each chunk runs each of the 100 PDFs 5 times. A CLI option is also added for local runs so that users can select a specific pdfpaint PDF to test. An additional issue with the subtest/pdf file name parsing is also fixed for this to work.

Differential Revision: https://phabricator.services.mozilla.com/D205824
2024-04-08 11:47:05 +00:00
florin.bilt 012a0bb963 Bug 1615258 - [perfdocs] Add test manifest searchfox link to test descriptions r=perftest-reviewers,sparky,jmaher,afinder
Differential Revision: https://phabricator.services.mozilla.com/D205060
2024-04-08 08:11:11 +00:00
Joel Maher 0d4582544d Bug 1889658 - Remove Windows XP/7 references from some test scripts. r=gbrown,perftest-reviewers,taskgraph-reviewers,sparky
Differential Revision: https://phabricator.services.mozilla.com/D206651
2024-04-08 01:45:49 +00:00
Sandor Molnar e135a31047 Backed out changeset bcbab342eed8 (bug 1889658) for causing wpt reftest failures CLOSED TREE 2024-04-06 02:01:53 +03:00
Joel Maher 0a92dab271 Bug 1889658 - Remove Windows XP/7 references from some test scripts. r=gbrown,perftest-reviewers,taskgraph-reviewers,sparky
Differential Revision: https://phabricator.services.mozilla.com/D206651
2024-04-05 20:47:41 +00:00
Alex Finder 67bd60c445 Bug 1888398 Disable unity-webgl on safari on macosx1015 r=perftest-reviewers,kshampur
Differential Revision: https://phabricator.services.mozilla.com/D206490
2024-04-03 15:43:46 +00:00
Cristian Tuns 77e58c81d0 Backed out changeset ace4b125b6e0 (bug 1615258) for causing py3 failures in test_manifestparser.py CLOSED TREE 2024-03-28 08:58:42 -04:00
florin.bilt 454d225cfa Bug 1615258 - [perfdocs] Add test manifest searchfox link to test descriptions r=perftest-reviewers,sparky,jmaher,afinder
Differential Revision: https://phabricator.services.mozilla.com/D205060
2024-03-28 12:37:04 +00:00
andrej 75361b871a Bug 1867837 - Remove chromium specific code from raptor, CI and locally. r=perftest-reviewers,kshampur
As chromium is no longer used and chromium as release is replacing it, we are removing all references to it to not confuse people

Differential Revision: https://phabricator.services.mozilla.com/D204821
2024-03-26 04:40:03 +00:00
Greg Mierzwinski c9f6950008 Bug 1878709 - Add, and update tp6-bench docs. r=kshampur,perftest-reviewers DONTBUILD
This patch adds some additional documentation about the tp6-bench test to the Raptor Browsertime docs. At the same time, the description given in the perfdocs config.yml file is updated to better reflect what the test currently does.

Differential Revision: https://phabricator.services.mozilla.com/D203821
2024-03-25 15:09:36 +00:00
andrej 1ac7a08138 Bug 1886168 - Run sp3 tests daily against p6 and s21 on geckoview. r=perftest-reviewers,sparky
In this bug we want to enable sp3 to run daily on the p6 and s21 on fenix, geckoview and chrome
Since chrome and fenix are already running regularly due to other patches this is just running geckoview regularly

Differential Revision: https://phabricator.services.mozilla.com/D205565
2024-03-25 14:52:31 +00:00
Greg Mierzwinski 40ee83bda3 Bug 1860111 - Remove talos windows10 ref hw tests, and platform configurations. r=jmaher,taskgraph-reviewers,perftest-reviewers,kshampur
This patch removes the windows 10 reference hardware talos tests, and removes the platform config for it. With talos tests disabled on the windows10 reference hardware, we can also remove any configuration steps that we have for that platform since there are no other tests running on those machines.

Differential Revision: https://phabricator.services.mozilla.com/D205080
2024-03-20 18:14:06 +00:00
Norisz Fay 6fad4f08ca Backed out changeset 9bad61381e60 (bug 1860111) for causing python failure on test_set_test_setting CLOSED TREE 2024-03-20 15:10:25 +02:00
Greg Mierzwinski c3a53cdb6c Bug 1860111 - Remove talos windows10 ref hw tests, and platform configurations. r=jmaher,taskgraph-reviewers,perftest-reviewers,kshampur
This patch removes the windows 10 reference hardware talos tests, and removes the platform config for it. With talos tests disabled on the windows10 reference hardware, we can also remove any configuration steps that we have for that platform since there are no other tests running on those machines.

Differential Revision: https://phabricator.services.mozilla.com/D205080
2024-03-20 12:21:37 +00:00
Greg Mierzwinski 23406a2ed5 Bug 1866047 - Remove --fenix flag from mach-try-perf. r=aglavic,perftest-reviewers,taskgraph-reviewers,bhearsum
This patch undoes the changes required for the `--fenix` flag. This will enable fenix tests to be scheduled without supplying an additional flag, and will be visible/schedule-able when only `--android` is supplied.

Differential Revision: https://phabricator.services.mozilla.com/D203649
2024-03-19 19:00:34 +00:00
Greg Mierzwinski 386afa9714 Bug 1881785 - Run fenix performance tests on mozilla-central, and autoland. r=aglavic,perftest-reviewers,taskgraph-reviewers,jcristau
This patch sets up the scheduling for fenix performance tests on mozilla-central, and autoland. All the same tests will be running across both branches. However, the youtube-playback tests will only be running on mozilla-central. It also removes fenix speedometer tests from the gpt cron since they'll be running on every commit.

Differential Revision: https://phabricator.services.mozilla.com/D202872
2024-03-19 19:00:34 +00:00
Greg Mierzwinski fc141a0669 Bug 1883397 - Update raptor-browsertime perfdocs with more recent information. r=aglavic,perftest-reviewers,taskgraph-reviewers,bhearsum DONTBUILD
This patch updates the Raptor Browsertime perfdocs with more recent information, and includes additional information about running performance tests on android APKs locally.

Differential Revision: https://phabricator.services.mozilla.com/D203791
2024-03-11 22:09:51 +00:00
florin.bilt 1b345d96fe Bug 1874193 - [perfdocs] Ensure raptor descriptions are added test documentation r=perftest-reviewers,afinder
Differential Revision: https://phabricator.services.mozilla.com/D202728
2024-03-11 09:45:28 +00:00
Jeff Muizelaar 08a98e1088 Bug 1884421 - Update Speedometer 3 repo to 8d67f28d0281ac4330f283495b7f48286654ad7d. r=mstange
https://github.com/WebKit/Speedometer/compare/64a5e7c0ba0f668834c0adf01b933e54d2b0d1f0...8d67f28d0281ac4330f283495b7f48286654ad7d

This includes:
jQuery workloads: use :nth-child() in a querySelector call in each it…

Differential Revision: https://phabricator.services.mozilla.com/D204075
2024-03-08 21:30:22 +00:00
Greg Mierzwinski b5717bcd40 Bug 1860111 - Run talos-webgl-gli tests on windows 11 ref hw. r=afinder,perftest-reviewers
This patch enables the talos-webgl-gli tests on the windows 11 reference hardware which will replace the windows 10 reference hardware.

Differential Revision: https://phabricator.services.mozilla.com/D203298
2024-03-06 17:47:25 +00:00
andrej 0e837b65e0 Bug 1882124 - Enable regular running of speedometer3 on the p6, and s21. r=perftest-reviewers,taskgraph-reviewers,kshampur,ahal
We are adding the ability to run chrome, fenix and custom-car regularly on mozilla central
We need data from the new android devices for speedometer3, so we are temporarily adding the ability to run custom-car, chrome-m and fenix on the new generation of devices we want to test on.
Only speedometer3 for now as that is all we have capacity for at the moment

Differential Revision: https://phabricator.services.mozilla.com/D202867
2024-03-05 18:20:16 +00:00
KS 835a0abaff Bug 1689537 - Update config files for MotionMark 1.3 benchmark. r=perftest-reviewers,aglavic
This patch updates the toml/yml files necessary to run motionmark 1.3.
An external repo model is used here rather than vendoring in-tree, as
this should be more advantageous going forward.
Brand new files (motionmark-1-3-<desktop/mobile>.toml) are added rather
than adding it to the existing motionmark animometer/htmlsuite config files.
This should be easier to maintain going forward (if for example we
decide to remove the previous motionmark). And also there has has been
so many changes since motionmark 1.0, it may be worth thinking of this
as a new benchmark entirely.

It is also worth noting that what was formerly known as
`motionmark-animometer` is now just `motionmark`. However htmlsuite is
still the same.

Differential Revision: https://phabricator.services.mozilla.com/D202531
2024-03-05 17:51:35 +00:00
Greg Mierzwinski 1747ac6b38 Bug 1882870 - Add documentation on perftest infrastructure. r=kshampur,perftest-reviewers
This patch adds a new perfdocs page for documentation about our perftest infrastructure. It includes information about try run wait times, along with a graph showing current pending tasks (refreshed hourly). It also includes information about how, and when requests for additional devices should be made.

Differential Revision: https://phabricator.services.mozilla.com/D203185
2024-03-05 13:24:53 +00:00
Greg Mierzwinski 9d8112ce0e Bug 1881862 - Re-record sina pageload test. r=perftest-reviewers,aglavic
This patch re-records the sina pageload test. It also adds some additional code to the recording script to attempt to remove the cookie banners.

Differential Revision: https://phabricator.services.mozilla.com/D202639
2024-02-27 16:27:29 +00:00
Greg Mierzwinski 5ddf5bb6a9 Bug 1880115 - Replace web-de test with bild-de. r=perftest-reviewers,kshampur
This patch replaces the web-de test with bild-de since the web-de test can no longer be re-recorded due to the full-page cookie banner that is shown there.

Differential Revision: https://phabricator.services.mozilla.com/D202226
2024-02-23 18:33:04 +00:00
andrej d042069536 Bug 1848400 - Upgrade M2s to use OSX 14 instead of OSX 13. r=perftest-reviewers,sparky
During the enabling of safaridriver we ran into user permission issues and as a result were unable to enable run safari tests on M2s
The issue we were encountering was `kernel: (Sandbox) Sandbox: com.apple.WebDriver.HTTPService(30714) deny(1) network-bind local:*:7055`.
This issue was precenting from webdriver from starting and the permission issue was caused by running a taskcluster-worker as system daemon vs. a per-user launch agent.

The issue was specific to OSX 13, and so it was decided to upgrade the machines from OSX 13 to OSX 14 to resolve the issue and give us a more modern OS to test speedometer against in general

Differential Revision: https://phabricator.services.mozilla.com/D202503
2024-02-23 14:52:12 +00:00
csevere e31ebcf25b Bug 1862687 Update documentation issue on tools/lint/perfdocs/templates/index.rst. r=sparky,perftest-reviewers DONTBUILD
Differential Revision: https://phabricator.services.mozilla.com/D201911
2024-02-15 16:21:24 +00:00
florin.bilt 9253fad63d Bug 1874961 - Run nightly-as-release speedometer3 on mozilla-central instead of autoland r=perftest-reviewers,sparky
Differential Revision: https://phabricator.services.mozilla.com/D201718
2024-02-14 14:38:49 +00:00
Greg Mierzwinski 650363136e Bug 1879598 - Disable talos/awsy on beta/release branches. r=perftest-reviewers,taskgraph-reviewers,ahal,kshampur DONTBUILD
This patch disable the talos/awsy tests on the beta/release branches. In another patch, these tests will be run on those branches through a cron instead.

Depends on D201250

Differential Revision: https://phabricator.services.mozilla.com/D201278
2024-02-12 18:29:00 +00:00
Greg Mierzwinski 067cebaec5 Bug 1873723 - Disable browsertime tests running on beta/release. r=perftest-reviewers,kshampur DONTBUILD
This patch disables browsertime tests running on beta/release through the run-on-projects settings. In a follow-up patch these will be run through a cron instead.

Differential Revision: https://phabricator.services.mozilla.com/D201249
2024-02-12 18:28:59 +00:00
andrej 1ea93a33cb Bug 1874900 - Add taskcluster configs and enable speedometer for the Samsung S21s. r=perftest-reviewers,afinder,sparky
As a part of our performance strategy we are expanding the devices we run performance tests on
In this patch we are adding the S21 taskcluster definitions and the ability to run speedometer and speedometer3 on those devices

Differential Revision: https://phabricator.services.mozilla.com/D198721
2024-02-12 13:00:04 +00:00
florin.bilt 3b8a33f0a5 Bug 1796627 - Update google-slides secondary_url from a document url to a slides presentation url r=perftest-reviewers,kshampur
Differential Revision: https://phabricator.services.mozilla.com/D201090
2024-02-12 11:00:55 +00:00