Make it easier to write a new performance test by adding documention on how to select a test framework as well as starter code.
Differential Revision: https://phabricator.services.mozilla.com/D289304
As a result of a mid-march update of chrome on lambdatest devices chrome navigation end tests regressed.
The numbers while valid do not reflect real-experiences of applink, the adjustment to the threshold error bring the numbers back to what real people experience.
Differential Revision: https://phabricator.services.mozilla.com/D295289
Third party accessibility services activate accessibility mode in both Fenix and Chrome, distorting the performance results and undermining the validity of measurements taken on devices intended for performance testing.
Differential Revision: https://phabricator.services.mozilla.com/D289197
Replace the HTTP/1.0 unencrypted Python server with an HTTP/2 over TLS Node.js
server (node:http2) for more realistic mobile startup performance testing.
Falls back to plain HTTP without root access.
Uses the existing in-tree HTTP/2 test certificates:
- netwerk/test/unit/http2-ca.pem: Self-signed CA ("HTTP2 Test CA"). Installed
on Android so Chrome recognizes the server cert as valid.
- testing/raptor/browsertime/utils/http2-cert.pem: Server certificate signed
by the CA, with localhost in its SANs. Sent to clients during the TLS
handshake.
- testing/raptor/browsertime/utils/http2-cert.key: Private key for the server
certificate, used by the Node.js server to complete the TLS handshake.
These paths are added to the perftest sparse profile so they are available in
CI checkouts. Fenix trusts the server via cert_override.txt written to the
GeckoView profile.
Differential Revision: https://phabricator.services.mozilla.com/D289666
On the pixel6 we keep getting errors related to security hub not responding at an incidence rate of around 10%.
After digging through the logs we keep getting some sort of fail from the com.google.android.apps.security.securityhub package.
The resolution was to clear the package data between runs, to avoid the issue occuring at all.
Differential Revision: https://phabricator.services.mozilla.com/D289336
Due to bug 2005825 process names may contain the tag
"_disable_art_image_". We should ignore this when displaying the
per-process measurements in the resource usage and applink startup
tests.
Differential Revision: https://phabricator.services.mozilla.com/D276741
Moved sorting into archive_files to ensure correct ordering when appending across multiple profiler runs for an easier view when looking at the profiles through the Firefox Profiler. Files are sorted numerically by index.
Differential Revision: https://phabricator.services.mozilla.com/D280312
It's possible to specify in a `tryselect` preset (e.g., the current definition of the `webgpu` preset) `rebuild: …`, but it's not possible to set it back to 1 from the CLI if that's desired.
Lower the limit for this flag from 2 to 1, so folks who would like a `rebuild` value in their preset don't have to do gymnastics.
Differential Revision: https://phabricator.services.mozilla.com/D279210
This now makes it so all variables in base_measures must exist before we enter the metrics.extend section, thereby removing all risk of a keyerror which has been intermittently causing issues
Differential Revision: https://phabricator.services.mozilla.com/D272063
The image analysis of the pixel6 was returning invalid numbers and not telling us how we are comparing to chrome properly
My best guess is that the pixel6 device compresses images more agressively, and as a result small static noise builds up and makes the error look bigger than it is
To resolve this I first converted all the images we compare to black and white, and then after taking the image difference apply a binary threshold filter to remove all the noise associated with compression
The results gives stable results on the pixel and all other phones without sacrificing accuracy
Differential Revision: https://phabricator.services.mozilla.com/D265490
This was a Claude associated change, but is virtually the same as what was done for mochitests in https://bugzilla.mozilla.org/show_bug.cgi?id=1982404.
Ideally these would be further split up. This is not possible in part because of how `PREFIX_BY_KIND` works, most more notably because we need to write out the `tests-by-manifest.json.gz` artifact for these tests -- and we need to generate all web platform tests in the same process we write that out in.
Differential Revision: https://phabricator.services.mozilla.com/D265673
As recorded in 1985798 we've seen an issue where the startup time has been over 2 min(when the test is expected to run well under 20 seconds)
This should prevent when this issue occurring again and from interfering in the comparison report data(from being a huge outlier).
Differential Revision: https://phabricator.services.mozilla.com/D264326
In this patch I am improving the image analysis capabilities by doing the following:
1) Tightening of the acceptable error for the image analysis: Each test results in different images which load in different ways, so the acceptable error bound should be per-test to accomodate different tests.
2) Compare images in linear not bisection method: The startup is complete much closer to the end and therefore it can be done with fewer comparisons if we just work backwards linearly
3) Removing frame completed number: Occasionally adb screen record drops frames, we have no way of telling this and as a result frame numbers are different on the screen vs what we count, I have removed the frame numbers from the file name, and instead do not crop the photo that is saved, that way we can still tell exactly what the completed frame is but without any off by 1/2 errors
Differential Revision: https://phabricator.services.mozilla.com/D260380
This adds a new section to Performance Testing in a Nutshell covering
the newly modified `geckoprofile` action task in Treeherder. This new
section explains step by step how users can retrigger performance tests
with custom Gecko Profiler parameters without having to submit a new
Try job with non-default settings.
Differential Revision: https://phabricator.services.mozilla.com/D257803
This patch adds the initial documentation for telemetry alerting. It includes most of the information needed, however certain parts like default settings or the monitoring sections are lacking since they depend on the protoyping phase. The tooling overview may also change, but should cover the tool in a general sense.
Differential Revision: https://phabricator.services.mozilla.com/D257067
The fix for the big number discrepancy was a result of us taking the difference rather than the absolute difference
Changing to absolute difference reduced the noise and does not generate the huge swings in applink startup newssite data(where network is restricted)
This was confirmed by using the artifacts listed in this comment:https://bugzilla.mozilla.org/show_bug.cgi?id=1972093#c5
Differential Revision: https://phabricator.services.mozilla.com/D253723
This patch adds some additional documentation to the perftest-in-a-nutshell docs to mention the usage of the WORKSFORME bug resolution option. It also slightly modifies the inactive alert process to mention this resolution option.
Differential Revision: https://phabricator.services.mozilla.com/D253381
Abhishek Nimalan