This patch updates our documents to include a section with instructions
on how a user can run a browsertime or raptor-browsertime test locally
with safari tech preview
Differential Revision: https://phabricator.services.mozilla.com/D209878
This patch adds the ability to run Raptor performance tests
(specifically just speedometer 3 for now) on Safari Technology Preview.
Differential Revision: https://phabricator.services.mozilla.com/D209611
This patch updates perfdocs with talos doc changes, and makes a modification to the perfdocs-verify task to trigger it on reviewbot again.
Differential Revision: https://phabricator.services.mozilla.com/D209228
This patch updates perfdocs with talos doc changes, and makes a modification to the perfdocs-verify task to trigger it on reviewbot again.
Differential Revision: https://phabricator.services.mozilla.com/D209228
This patch adds a couple sentences to our standard-workflow docs for mach-try-perf to explain the use replicates button.
Differential Revision: https://phabricator.services.mozilla.com/D208877
Grab the correct folder after extracting the x64 chromedrivers.
Previously we were always using the 32 bit chromedrivers in CI and only
recently added the 64 bit chromedrivers for win x64 platforms.
Additionally, since we don't make use of the 32 bit windows platform
anymore, it is removed for the chrome platform.
Differential Revision: https://phabricator.services.mozilla.com/D208001
This patch adds the ability to document metrics dynamically. The metrics are documented in the config YAML file for each framework, and then defined in a metrics.rst file. Two things are expected to be found in the framework docs now if they want to use dynamic metric documentation:
* A metrics.rst file that contains a `{metrics_documentation}` entry somewhere to insert the built documentation.
* A `{metrics_rst_name}` entry somewhere in the top-level/index.rst file so that a link can be produced to the metrics documentation.
The metrics documentation is specific to each framework. See the schema for the metrics for what is expected in this field. At the same time, the dynamic documentation for Raptor is generated here. Each metric in Raptor contains a list of all the tests that use (alert on) the specific metric. The tests link back to the test definition, and the metrics defined in the tests also link to the metric definitions.
Differential Revision: https://phabricator.services.mozilla.com/D206950
This patch enables the ability to run tp7 tests in raptor-browsertime.
At the moment these tests will not run on CI, but are available to run
locally. This is merely preliminary work on desktop that will help when
shifting focus to the android platform.
Differential Revision: https://phabricator.services.mozilla.com/D206826
This bug re-records a subset of some sites that are potentially causing
an intermittent issue (Bug 1774954)
Here, only `amazon`, `imgur`, and `instagram` are re-recorded. Other
sites may be re-recorded in a follow up bug.
Differential Revision: https://phabricator.services.mozilla.com/D206572
This patch adds a set of tasks for the talos pdfpaint test in CI. It'll run with, and without the swr variant on linux, windows, and mac. The number of chunks are defined manually in a taskcluster transform where the test is chunked.
Differential Revision: https://phabricator.services.mozilla.com/D205825
This patch modifies the pdfpaint test to run more pdfs that are found in the Mozilla pdf.js repository. The pdfpaint test is also moved to it's own suite due to the number of PDFs now being tested. These PDFs are pulled in locally from a toolchain task called talos-pdfs. The *ignore files are modified since the pdfpaint folder now contains a symbolic link to the local PDFs that should not be commit in-tree.
To handle running the large number of PDFs, chunking is added to the test with the chunk size being 100 PDFs. Each chunk runs each of the 100 PDFs 5 times. A CLI option is also added for local runs so that users can select a specific pdfpaint PDF to test. An additional issue with the subtest/pdf file name parsing is also fixed for this to work.
Differential Revision: https://phabricator.services.mozilla.com/D205824
As chromium is no longer used and chromium as release is replacing it, we are removing all references to it to not confuse people
Differential Revision: https://phabricator.services.mozilla.com/D204821
This patch adds some additional documentation about the tp6-bench test to the Raptor Browsertime docs. At the same time, the description given in the perfdocs config.yml file is updated to better reflect what the test currently does.
Differential Revision: https://phabricator.services.mozilla.com/D203821
In this bug we want to enable sp3 to run daily on the p6 and s21 on fenix, geckoview and chrome
Since chrome and fenix are already running regularly due to other patches this is just running geckoview regularly
Differential Revision: https://phabricator.services.mozilla.com/D205565
This patch removes the windows 10 reference hardware talos tests, and removes the platform config for it. With talos tests disabled on the windows10 reference hardware, we can also remove any configuration steps that we have for that platform since there are no other tests running on those machines.
Differential Revision: https://phabricator.services.mozilla.com/D205080
This patch removes the windows 10 reference hardware talos tests, and removes the platform config for it. With talos tests disabled on the windows10 reference hardware, we can also remove any configuration steps that we have for that platform since there are no other tests running on those machines.
Differential Revision: https://phabricator.services.mozilla.com/D205080
This patch undoes the changes required for the `--fenix` flag. This will enable fenix tests to be scheduled without supplying an additional flag, and will be visible/schedule-able when only `--android` is supplied.
Differential Revision: https://phabricator.services.mozilla.com/D203649
This patch sets up the scheduling for fenix performance tests on mozilla-central, and autoland. All the same tests will be running across both branches. However, the youtube-playback tests will only be running on mozilla-central. It also removes fenix speedometer tests from the gpt cron since they'll be running on every commit.
Differential Revision: https://phabricator.services.mozilla.com/D202872
This patch updates the Raptor Browsertime perfdocs with more recent information, and includes additional information about running performance tests on android APKs locally.
Differential Revision: https://phabricator.services.mozilla.com/D203791
This patch enables the talos-webgl-gli tests on the windows 11 reference hardware which will replace the windows 10 reference hardware.
Differential Revision: https://phabricator.services.mozilla.com/D203298
We are adding the ability to run chrome, fenix and custom-car regularly on mozilla central
We need data from the new android devices for speedometer3, so we are temporarily adding the ability to run custom-car, chrome-m and fenix on the new generation of devices we want to test on.
Only speedometer3 for now as that is all we have capacity for at the moment
Differential Revision: https://phabricator.services.mozilla.com/D202867
This patch updates the toml/yml files necessary to run motionmark 1.3.
An external repo model is used here rather than vendoring in-tree, as
this should be more advantageous going forward.
Brand new files (motionmark-1-3-<desktop/mobile>.toml) are added rather
than adding it to the existing motionmark animometer/htmlsuite config files.
This should be easier to maintain going forward (if for example we
decide to remove the previous motionmark). And also there has has been
so many changes since motionmark 1.0, it may be worth thinking of this
as a new benchmark entirely.
It is also worth noting that what was formerly known as
`motionmark-animometer` is now just `motionmark`. However htmlsuite is
still the same.
Differential Revision: https://phabricator.services.mozilla.com/D202531
This patch adds a new perfdocs page for documentation about our perftest infrastructure. It includes information about try run wait times, along with a graph showing current pending tasks (refreshed hourly). It also includes information about how, and when requests for additional devices should be made.
Differential Revision: https://phabricator.services.mozilla.com/D203185
This patch re-records the sina pageload test. It also adds some additional code to the recording script to attempt to remove the cookie banners.
Differential Revision: https://phabricator.services.mozilla.com/D202639
This patch replaces the web-de test with bild-de since the web-de test can no longer be re-recorded due to the full-page cookie banner that is shown there.
Differential Revision: https://phabricator.services.mozilla.com/D202226
During the enabling of safaridriver we ran into user permission issues and as a result were unable to enable run safari tests on M2s
The issue we were encountering was `kernel: (Sandbox) Sandbox: com.apple.WebDriver.HTTPService(30714) deny(1) network-bind local:*:7055`.
This issue was precenting from webdriver from starting and the permission issue was caused by running a taskcluster-worker as system daemon vs. a per-user launch agent.
The issue was specific to OSX 13, and so it was decided to upgrade the machines from OSX 13 to OSX 14 to resolve the issue and give us a more modern OS to test speedometer against in general
Differential Revision: https://phabricator.services.mozilla.com/D202503
This patch disable the talos/awsy tests on the beta/release branches. In another patch, these tests will be run on those branches through a cron instead.
Depends on D201250
Differential Revision: https://phabricator.services.mozilla.com/D201278
This patch disables browsertime tests running on beta/release through the run-on-projects settings. In a follow-up patch these will be run through a cron instead.
Differential Revision: https://phabricator.services.mozilla.com/D201249
As a part of our performance strategy we are expanding the devices we run performance tests on
In this patch we are adding the S21 taskcluster definitions and the ability to run speedometer and speedometer3 on those devices
Differential Revision: https://phabricator.services.mozilla.com/D198721