This patch adds a new tool that runs a comparison between two, or more revisions to detect performance changes. Some changes are made to accommodate this new tool alongside the side-by-side tool. The tests for the detector coding are found in another patch in the series. A mozperftest-tools update to 0.2.5 is required for this change.
There is also a CI task that is added in this patch. It's setup in the mach try perf patch in this series, which also has more information.
Differential Revision: https://phabricator.services.mozilla.com/D172282
This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.
The `./mach try perf` command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the `./mach try perf` command to select the appropriate tests.
Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.
Differential Revision: https://phabricator.services.mozilla.com/D172435
This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.
The `./mach try perf` command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the `./mach try perf` command to select the appropriate tests.
Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.
Differential Revision: https://phabricator.services.mozilla.com/D172435
This patch adds the ability to run manual logins for our websites since it can be simpler, and quicker in some cases. At the same time, a bug with the options handling is fixed.
Differential Revision: https://phabricator.services.mozilla.com/D164590
This results in some changes from our current `isort` configuration. I'm
unclear if it's because ruff isn't at 100% parity with isort, they choose
different defaults or if I missed some configuration.
Either way, the changes all look reasonable to me (see child commit), so I'm
inclined to just accept the new import format it imposes.
Differential Revision: https://phabricator.services.mozilla.com/D172348
This results in some changes from our current `isort` configuration. I'm
unclear if it's because ruff isn't at 100% parity with isort, they choose
different defaults or if I missed some configuration.
Either way, the changes all look reasonable to me (see child commit), so I'm
inclined to just accept the new import format it imposes.
Differential Revision: https://phabricator.services.mozilla.com/D172348
This patch fixes an issue in our perftestetl.py which was causing our coverage percent to drop "artificially". This is happening because there's a bit of code that's unreachable because of an exception that would always be triggered before it. This patch removes the unreachable code, and replaces it with an exception.
Differential Revision: https://phabricator.services.mozilla.com/D172281
This patch updates the existing tests so that they're inline with the new default of providing no summary value at the suite level. A new test is also added to test out custom transforms.
Differential Revision: https://phabricator.services.mozilla.com/D171613
This patch removes the default summary value for perfherder suites. Afterwards, no summary value will be produced unless a new Transformer is provided, or the existing ones are modified.
The patch goes further and creates a path through which we can let users modify the suites, and subtest in any way they want. This is done by adding two new methods that can be added to the transformer being used on the data. Some changes also needed to be made to the MetricsStorage so that we keep track of the transformer. They can be specified on a per-subtest basis some day, but right now it applies a single transformer to all subtests of a given suite/data-type.
Differential Revision: https://phabricator.services.mozilla.com/D171612
This patch adds an FAQ (Frequently Asked Questions) section to the mach try perf docs. It also does a small cleanup to move fxrecord into the `testing/performance` folder, re-organize the linting configuration file, fix file naming, and captializes the `mozperftest` and `fxrecord` title names in the side-bar. Lastly, it adds a warning to the `mozperftest` docs to direct people who make it there to the `mach try perf` page.
Differential Revision: https://phabricator.services.mozilla.com/D167555
What are we doing:
- Resolving a few bugs/user requests
Issues being addressed:
- Resolved issue where if the WPT_key.txt file is not available locally it does not affect running ./mach perftest-test
- Added section to WPT where we display the amount of tests we have remaining
- Altered the request_with_timeout function, to better handle requests
Differential Revision: https://phabricator.services.mozilla.com/D155268
The older versions don't have prebuilt wheels for python 3.9 and newer.
Unfortunately, the latest versions don't support python 3.7 and older,
so keep the older versions for those.
Differential Revision: https://phabricator.services.mozilla.com/D152246
A week ago we received a notification that we had a test that the WPT chrome tests were perma failing on [[ https://bugzilla.mozilla.org/show_bug.cgi?id=1773621 | bugzilla ]]
After going through the fail logs I realized it was because of website "panda.tv" directing to a unable to connect page message, after some digging it was not returning proper data because panda.tv has not been a company since March 2019(bankruptcy filing), why this only is causing an issue now I believe is because of some kind of update from WPT as I can see a noticeable UI difference on the test results page from before and after the failures started.
My resolution was to remove Panda.tv from our test list and that seems to have resolved the issue.
I also updated the error message to display which website is causing the issue so that if this happens again I don't need to go through each and every webpagetest result to know which of the 40 websites are having an issue.
Differential Revision: https://phabricator.services.mozilla.com/D149642
The patch aims to improve the `_should_install()` method for installing browsertime in mozperftest.
Here, the same approach used in `testing/raptor/mach_commands.py` is used to address potential issues (e.g. KeyError)
that one may encounter when trying to install from the `package.json` through this approach.
Differential Revision: https://phabricator.services.mozilla.com/D148565
This next patch in the series utilizes the same login-logic in Mozperftest and makes it available to the `pageload_test` method so that we can now automate the logging into of accounts during perftest recordings.
Additional logic is also added to account for if the site requires login, if we are running on CI or locally (and if on CI, accounting for the SCM level), and removal of the verbose flags so secrets do not leak.
Differential Revision: https://phabricator.services.mozilla.com/D147775
* There is no `--duration` argument, but there //is// `--durations`
* `pytest` behaves better when called via its main entry points (either
`bin/pytest`, or `-m pytest`)
Differential Revision: https://phabricator.services.mozilla.com/D143609
All commands declaring a virtualenv will have them activated before the
command executes. Removes all now-redundant manual activations of
declared virtualenvs.
Commands that don't declare a virtualenv will still implicitly be
associated with the "common" virtualenv, but unlike explicit
virtualenv declarations it'll have to be activated manually, just
like it was before this patch.
To smooth the migration with existing usages, virtualenv activation
behaviour was changed slightly: if attempting to activate a new
virtualenv, but the source venv is already command venv, then raise an
exception. (In the future, we should improve testability of
virtualenv scaffolding logic so that tests can be added for this
sort of thing.) This did cause some issues with some tests, which
will be solved more cleanly with bug 1724273. In the meantime,
minimal modifications were made to failing tests to keep them green:
* `test_command_line.py` was activating the `common` virtualenv so
that it could install `mozproxy`, and use its CLI. Instead, I
modified the test to use `mozproxy` using the "module" interface
(`python -m mozproxy ...`). At that point, `MozbuildObject` was
unnecessary and usages were replaced with simpler variants.
* `test_vendor.py` needed its explicit `activate_virtualenv()` call
patched out. It still needs to use a virtualenv's Python
executable, but due to `sys.executable` now being kept up-to-date
as of bug 1717051, it could be used directly.
Differential Revision: https://phabricator.services.mozilla.com/D122892
