Magic-Mirror/gecko-dev
1
0
Fork 0
mirror of https://github.com/mozilla/gecko-dev.git synced 2024-11-26 22:32:46 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Bug 1928416 - Run jetstream2 benchmark on apple silicon + android. r=perftest-reviewers,fbilt

Browse Source 
Currently lacking ARM coverage for the jetstream2 test, so this patch
enables it on the apple silicon and android A55 device.

Differential Revision: https://phabricator.services.mozilla.com/D228239
This commit is contained in:
KS 2024-11-20 17:34:31 +00:00
parent 6025b34a2d
commit 4a85fa6154
8 changed files with 397 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
taskcluster/kinds/test/browsertime-desktop.yml
View File

@ -350,11 +350,14 @@ browsertime-benchmark:
by-app:
safari-tp:
- [speedometer3, sp3]
# Bug 1931699, jetstream permafailing
# - [jetstream2, js2]
default:
- [speedometer, sp]
- [speedometer3, sp3]
- [motionmark-1-3, mm-1.3]
- [motionmark-htmlsuite-1-3, mm-h-1.3]
- [jetstream2, js2]
# Bug 1909894, windows 10 + sp3 temporarily
windows10-64.*shippable.*-qr.*:
by-app:
@ -442,7 +445,15 @@ browsertime-benchmark:
default:
by-subtest:
unity-webgl: 3
jetstream2:
by-test-platform:
macosx1400.*shippable.*:
by-app:
firefox: 2
default: 3
default: 2
default: 2
run-on-projects:
by-subtest:
speedometer3:

20
taskcluster/kinds/test/browsertime-mobile.yml
View File

@ -478,6 +478,26 @@ browsertime-benchmark-motionmark-1-3:
geckoview: 2
default: 3
browsertime-benchmark-jetstream2:
description: Raptor (browsertime) jetstream 2 benchmark test
run-without-variant: false
raptor:
apps: [fenix, chrome-m, cstm-car-m]
subtests:
- [jetstream2, js]
run-on-projects:
by-test-platform:
android-hw-a55.*shippable.*:
by-app:
fenix: [mozilla-central]
default: []
default: []
treeherder-symbol: Btime()
tier:
by-app:
fenix: 2
default: 3
browsertime-benchmark-unity-webgl-mobile:
description: Browsertime Unity WebGL on Android
raptor:

1
taskcluster/kinds/test/test-sets.yml
View File

@ -518,6 +518,7 @@ android-hw-browsertime:
- browsertime-youtube-playback-mobile
- browsertime-benchmark-unity-webgl-mobile
- browsertime-benchmark-motionmark-1-3
- browsertime-benchmark-jetstream2
- browsertime-video-playback-latency-mobile
android-hw-browsertime-profiling:

1
taskcluster/kinds/toolchain/misc.yml
View File

@ -266,7 +266,6 @@ android-custom-car:
target_os="android",
is_android=true,
is_high_end_android=true,
use_v8_context_snapshot=true,
v8_symbol_level=0
]
toolchain-artifact: public/build/car_android.tar.zst

344
testing/perfdocs/generated/raptor.rst
View File

@ -459,7 +459,7 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
**Owner**: :jandem and SpiderMonkey Team
* **alert threshold**: 2.0
* **apps**: firefox, chrome, safari
* **apps**: firefox, chrome, safari, custom-car
* **expected**: pass
* **expose browser profiler**: true
* **gecko profile interval**: 1
@ -477,6 +477,48 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
* **unit**: score
* **Test Task**:
.. list-table:: **test-android-hw-a55-14-0-aarch64-shippable/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-jetstream2-chrome-m-jetstream2-nofis**
- ❌
- ❌
- ❌
- ❌
* - **browsertime-benchmark-jetstream2-cstm-car-m-jetstream2-nofis**
- ❌
- ❌
- ❌
- ❌
* - **browsertime-benchmark-jetstream2-fenix-jetstream2-nofis**
- ✅
- ❌
- ❌
- ❌
.. list-table:: **test-android-hw-a55-14-0-aarch64/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-jetstream2-fenix-jetstream2-nofis**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-linux1804-64-clang-trunk-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
@ -599,6 +641,306 @@ Standard benchmarks are third-party tests (i.e. Speedometer) that we have integr
- ❌
.. list-table:: **test-macosx1400-64-shippable-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-chrome-jetstream2**
- ❌
- ❌
- ❌
- ❌
* - **browsertime-benchmark-custom-car-jetstream2**
- ❌
- ❌
- ❌
- ❌
* - **browsertime-benchmark-firefox-jetstream2**
- ✅
- ✅
- ❌
- ❌
* - **browsertime-benchmark-safari-jetstream2**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-windows11-64-nightlyasrelease-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-firefox-jetstream2**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-windows11-64-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-firefox-jetstream2**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-windows11-64-shippable-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-chrome-jetstream2**
- ❌
- ❌
- ❌
- ❌
* - **browsertime-benchmark-firefox-jetstream2**
- ✅
- ✅
- ❌
- ❌
**Owner**: :jandem and SpiderMonkey Team
* **alert threshold**: 2.0
* **apps**: fenix, chrome-m, cstm-car-m
* **expected**: pass
* **expose browser profiler**: true
* **gecko profile interval**: 1
* **link searchfox**: `<https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/benchmarks/jetstream2-mobile.toml#16>`__
* **lower is better**: false
* **page cycles**: 4
* **page timeout**: 2000000
* **repository**: https://github.com/mozilla/perf-automation
* **repository path**: benchmarks/JetStream2
* **repository revision**: 61332db584026b73e37066d717a162825408c36b
* **subtest lower is better**: false
* **subtest unit**: score
* **test url**: `<http://\<host\>:\<port\>/index.html?raptor>`__
* **type**: benchmark
* **unit**: score
* **Test Task**:
.. list-table:: **test-android-hw-a55-14-0-aarch64-shippable/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-jetstream2-chrome-m-jetstream2-nofis**
- ❌
- ❌
- ❌
- ❌
* - **browsertime-benchmark-jetstream2-cstm-car-m-jetstream2-nofis**
- ❌
- ❌
- ❌
- ❌
* - **browsertime-benchmark-jetstream2-fenix-jetstream2-nofis**
- ✅
- ❌
- ❌
- ❌
.. list-table:: **test-android-hw-a55-14-0-aarch64/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-jetstream2-fenix-jetstream2-nofis**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-linux1804-64-clang-trunk-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-firefox-jetstream2**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-linux1804-64-nightlyasrelease-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-firefox-jetstream2**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-linux1804-64-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-firefox-jetstream2**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-linux1804-64-shippable-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-chrome-jetstream2**
- ❌
- ❌
- ❌
- ❌
* - **browsertime-benchmark-firefox-jetstream2**
- ✅
- ✅
- ❌
- ❌
.. list-table:: **test-macosx1015-64-nightlyasrelease-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-firefox-jetstream2**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-macosx1015-64-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-firefox-jetstream2**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-macosx1015-64-shippable-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-firefox-jetstream2**
- ✅
- ✅
- ❌
- ❌
* - **browsertime-benchmark-safari-jetstream2**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-macosx1400-64-shippable-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1
* - **Test Name**
- mozilla-central
- autoland
- mozilla-release
- mozilla-beta
* - **browsertime-benchmark-chrome-jetstream2**
- ❌
- ❌
- ❌
- ❌
* - **browsertime-benchmark-custom-car-jetstream2**
- ❌
- ❌
- ❌
- ❌
* - **browsertime-benchmark-firefox-jetstream2**
- ✅
- ✅
- ❌
- ❌
* - **browsertime-benchmark-safari-jetstream2**
- ❌
- ❌
- ❌
- ❌
.. list-table:: **test-windows11-64-nightlyasrelease-qr/opt**
:widths: 30 15 15 15 15
:header-rows: 1

2
testing/raptor/raptor/raptor.toml
View File

@ -7,6 +7,8 @@
["include:tests/benchmarks/jetstream2.toml"]
["include:tests/benchmarks/jetstream2-mobile.toml"]
["include:tests/benchmarks/matrix-react-bench.toml"]
["include:tests/benchmarks/motionmark-1-3-desktop.toml"]

19
testing/raptor/raptor/tests/benchmarks/jetstream2-mobile.toml Normal file
View File

@ -0,0 +1,19 @@
[DEFAULT]
alert_threshold = 2.0
apps = "fenix, chrome-m, cstm-car-m"
gecko_profile_interval = 1
expose_browser_profiler = true
lower_is_better = false
owner = ":jandem and SpiderMonkey Team"
page_cycles = 4
page_timeout = 2000000
subtest_lower_is_better = false
subtest_unit = "score"
test_url = "http://<host>:<port>/index.html?raptor"
type = "benchmark"
unit = "score"
["jetstream2"]
repository = "https://github.com/mozilla/perf-automation"
repository_revision = "61332db584026b73e37066d717a162825408c36b"
repository_path = "benchmarks/JetStream2"

2
testing/raptor/raptor/tests/benchmarks/jetstream2.toml
View File

@ -1,6 +1,6 @@
[DEFAULT]
alert_threshold = 2.0
apps = "firefox, chrome, safari"
apps = "firefox, chrome, safari, custom-car"
gecko_profile_interval = 1
expose_browser_profiler = true
lower_is_better = false