Right now setup_perftest_test_date adds --test-date yesterday to all perftest
runs. we want that only for the ones doing batches
Differential Revision: https://phabricator.services.mozilla.com/D81562
This patch adds a `--test-date` option to mozperftest that lets us specify the date that we are testing on. This lets us trigger multi-commit tests on past dates, and also ensures that retriggers of the multi-commit tests use the same day that they were pushed on (giving us reproducibility).
Differential Revision: https://phabricator.services.mozilla.com/D80173
This patch fixes the tests in testing/performance/ that hadn't been updated to
the new API, as well as fixing a couple issues with the `--proxy` option:
* `policy.json` wasn't being logged correctly, since mozlog was treating the
data as a format string
* Proxy prefs weren't being set correctly for the profile
Differential Revision: https://phabricator.services.mozilla.com/D79937
This patch fixes the tests in testing/performance/ that hadn't been updated to
the new API, as well as fixing a couple issues with the `--proxy` option:
* `policy.json` wasn't being logged correctly, since mozlog was treating the
data as a format string
* Proxy prefs weren't being set correctly for the profile
Differential Revision: https://phabricator.services.mozilla.com/D79937
This patch adds a prefix field to the mozperftest perfherder data. This is a non-standard data field so the performance-artifact-schema.json file isn't modified to include it.
Differential Revision: https://phabricator.services.mozilla.com/D79296
This patch fixes the test name being used for the App-link test in CI (regression). The patch also adds a new feature to ScriptInfo which checks to make sure that the test doesn't have any extra unknown keys in it. If it does, a failure is raised. This change is to attempt to help prevent these kinds of regressions from happening again. A new test is added for this functinality as well.
Differential Revision: https://phabricator.services.mozilla.com/D78516
This patch fixes how the system and browser/test layers are run. With this fix, the system layer no longer fully runs its setup and teardown stages before the browser layer has started and finished. Now the setup/teardown happens before/after the full test layer run.
Depends on D78016
Differential Revision: https://phabricator.services.mozilla.com/D78128
This patch adds a transformer that is capable of extracting times/entries or durations for logcat logs. Here, we also implement the ability for users to pass options to the transformers so that they can be more generic (through transformer-options). The logcat transformer makes use of this, and also shows how a custom function can be used within transformers. A new test file is added for the transformer, as well as a test in test_perfherder.py so we can test the full pipeline.
Differential Revision: https://phabricator.services.mozilla.com/D77094
This patch surfaces the timeout value for ADBDevice.
It also adds the ability to run a single test with
mach perftest-test and a new --skip-lint option
to skip black/flake8
Differential Revision: https://phabricator.services.mozilla.com/D76791
This patch fixes a bug where a PosixPath is attempted to be used rather than a string path. There's also a regression in the coverage test run in `./mach perftest-test` on some platforms that is fixed by this patch (the attrs module being used by pytest is too old, so we update during setup). Lastly, a regression test is added to the browsertime tests to ensure that the command only contains knowingly compatible types.
Depends on D76713
Differential Revision: https://phabricator.services.mozilla.com/D76599
