This patch adds some more documentation to the perftest in a nutshell docs. This includes the regression workshop docs, some information about compare view, and some information about getting profiles from our tests.
Differential Revision: https://phabricator.services.mozilla.com/D219187
This patch adds a new background resource test for chrome release. The existing background resource test is modified to handle the chrome binary instead of building a separate test for it.
Differential Revision: https://phabricator.services.mozilla.com/D217123
This patch modifies the background-resource (formerly background-cpu) android test to collect resource information at distinct times rather than only at the end of the test run.
The collection script is split in two here for cpu, and memory. The python parsing script is also modified to be able to gather the resources from multiple files, and the name is changed to better reflect what it does.
This patch also renames the test to better reflect that it's a resource usage test and not just a cpu test.
Differential Revision: https://phabricator.services.mozilla.com/D217097
This patch adds some documentation about the --tests option, and also updates a bit of the other documentation for mach-try-perf.
Differential Revision: https://phabricator.services.mozilla.com/D216934
This patch adds a new location for a performance testing guide called "Performance Testing in a Nutshell". It'll contain an overview of all the information needed for understanding performance testing at Mozilla.
Differential Revision: https://phabricator.services.mozilla.com/D214282
As a result of their move from aws to google cloud webpagetest has updated the keys of the testing centre's location ids
Furthermore, I removed timeToContentfulPaint as it was not being returned from FF intermittently(most likely due to the change of services messing with things) rather than diagnosing the intermittent
Differential Revision: https://phabricator.services.mozilla.com/D214241
This patch adds a new background resource usage test that will run through mozperftest. The test starts the Fenix application, and then backgrounds it for 10 minutes to measure the CPU, and memory usage throughout that time. The hook is used to setup the xmlstarlet tool.
Differential Revision: https://phabricator.services.mozilla.com/D212630
As a result of the mono-repo change we have a very easy way to run the startup tests on each commit vs on a nightly build.
This change allows us to push a try job, build the fenix/focus/geckoview_example apk and test the startup impact!
Differential Revision: https://phabricator.services.mozilla.com/D208240
As a result of the mono-repo change we have a very easy way to run the startup tests on each commit vs on a nightly build.
This change allows us to push a try job, build the fenix/focus/geckoview_example apk and test the startup impact!
Differential Revision: https://phabricator.services.mozilla.com/D208240
This patch adds a couple sentences to our standard-workflow docs for mach-try-perf to explain the use replicates button.
Differential Revision: https://phabricator.services.mozilla.com/D208877
This patch introduces an ongoing tp7 desktop site list for pageload
testing. The following sites are added: `docomo`, `openai`, `samsung`,
`tiktok`, `weather`. This is merely a preliminary set of samples which
we may not even keep. This is more so preliminary work before focusing
on tp7 mobile.
Differential Revision: https://phabricator.services.mozilla.com/D206825
This bug re-records a subset of some sites that are potentially causing
an intermittent issue (Bug 1774954)
Here, only `amazon`, `imgur`, and `instagram` are re-recorded. Other
sites may be re-recorded in a follow up bug.
Differential Revision: https://phabricator.services.mozilla.com/D206572
This patch undoes the changes required for the `--fenix` flag. This will enable fenix tests to be scheduled without supplying an additional flag, and will be visible/schedule-able when only `--android` is supplied.
Differential Revision: https://phabricator.services.mozilla.com/D203649
The android startup script uses hg logs to get the date, this was failing for users using git cinnibar as they did not have hg installed
This patch adds compatability to the code that allows git cinnibar users without hg to run the android startup script
Differential Revision: https://phabricator.services.mozilla.com/D204562
This patch adds a new perfdocs page for documentation about our perftest infrastructure. It includes information about try run wait times, along with a graph showing current pending tasks (refreshed hourly). It also includes information about how, and when requests for additional devices should be made.
Differential Revision: https://phabricator.services.mozilla.com/D203185
This patch re-records the sina pageload test. It also adds some additional code to the recording script to attempt to remove the cookie banners.
Differential Revision: https://phabricator.services.mozilla.com/D202639
This patch replaces the web-de test with bild-de since the web-de test can no longer be re-recorded due to the full-page cookie banner that is shown there.
Differential Revision: https://phabricator.services.mozilla.com/D202226
This patch fixes an issue where multiple tests for the same app were conflicting with each other, preventing us from selecting which one we want to run. The conflicting tests were removed from the live sites list, and a separate issue related to the app name parsing from the manifest is also resolved here.
Differential Revision: https://phabricator.services.mozilla.com/D199892
The issue we were encountering here was a missing metric for canva.com intermittently occuring
That ocured because of the location of that metric(first contentful paint) in the returned json moved locations for just canva.com, and so canva.com was removed whilst the problem debugged. Afterwards it was discovered that this issue was covering up another intermittent about json decoding error which was in turn resolved by removing code which removed "www." from links used with python requests.
Differential Revision: https://phabricator.services.mozilla.com/D199238
This patch adds a flag to enable fenix in mach try perf. Currently, it's enabled by default however it's not clear that it uses the same build in the base/new try runs. This patch adds some documentation to the FAQ for this, as well as some information in the `--help` message.
At the same time, this patch adds some code to handle apps requested with `--apps` better. Previously, for example, if fenix was requested, the catch-all/base category would capture geckoview tasks as well, but it wasn't clear to the user that this would happen. Now, if an app is requested, that is the only app that will be scheduled, assuming the "default" app has a negation. The notion of a "default" app is also introduced here for any app that has no restriction method defined.
Finally, some extra changes are made to the tests for the fzf side effects to make it clear that the number of side effects must be greater than or equal to the number of fzf calls. This should prevent others from having to determine why their tests are failing on a seemingly unrelated change as the error message we receive is very unclear when this occurs.
Differential Revision: https://phabricator.services.mozilla.com/D194301
This patch adds documentation to the BasePerfComparator, and also adds some documentation for Comparators in general to the mach-try-perf Firefox source docs.
Differential Revision: https://phabricator.services.mozilla.com/D177760
Julien Wajsberg