They are not doing anything as try will choose jobs anyway.
And this is misleading for developers on what it is doing.
Differential Revision: https://phabricator.services.mozilla.com/D75490
The bugbug_debug algorithm wasn't a very high performer, but combining it with
'disperse' may yield some interesting results.
Differential Revision: https://phabricator.services.mozilla.com/D75401
This feature A) scheduled way too many tasks, and B) won't be that useful anyway once we
switch to manifest-scheduling.
Differential Revision: https://phabricator.services.mozilla.com/D75399
If a task has explicitly specified artifact paths, don't additionally specify
the default paths. If the task has private artifacts, having a directory
that uploads public artifacts seems like an attractive nuissance.
Differential Revision: https://phabricator.services.mozilla.com/D74200
The toolchain transform rarely changes in ways that impact the generated toolchain. but does occasionally change in ways that don't. Stop including the digest of the transform module, to avoid needlessly rebuilding toolchains as often. Given that the cache route is versioned, we can bump the version if the transform does change in a way that requires rebuilding.
Differential Revision: https://phabricator.services.mozilla.com/D74199
To help support KaiOS, we are running the kaios project branch in automation, with a
different trust domain. Allow configuring worker aliases using `trust-domain`, so
that they worker aliases do not need to be adjusted on that branch.
Differential Revision: https://phabricator.services.mozilla.com/D75240
These files are ALWAYS ignored and are never relevant when hashing the state of a working tree. It would be better overall to not consult the filesystem directly and go through the VCS to ensure we never try to hash any ignored files, but these files seem to be the main stumbling block and the primary cause of bugs like bug 1636797, so this is a fine stopgap in the meantime.
Differential Revision: https://phabricator.services.mozilla.com/D75126
Changes:
While the ubuntu1804 migration work was taking place, `pulseaudio` related pieces of code were sprinkled across a wide variety of modules.
Now that things have settled down, remove and streamline `pulseaudio` initialization so that there are less places for this to go wrong.
Differential Revision: https://phabricator.services.mozilla.com/D74809
Changes:
Perform chunking of the web-platform-test in the decision task, using a runtime-weighted strategy.
Valid web-platform-tests will have its here attribute extracted to build a dictionary of test paths keyed by the top-level directory of the test path.
Once the dictionary is built, the keys are compared against test runtime data extracted from ActiveData in descending order. For all keys where test runtime information exists, chunk from the longest running key.
For remaining keys, equally distribute among all chunks.
Differential Revision: https://phabricator.services.mozilla.com/D71417
Changes:
Discontinues the installation of `fontconfig:i386` package in favor of the x64 variant.
Also, use a newer base image for ubuntu1804.
Depends on D74809.
Differential Revision: https://phabricator.services.mozilla.com/D74810
Changes:
While the ubuntu1804 migration work was taking place, `pulseaudio` related pieces of code were sprinkled across a wide variety of modules.
Now that things have settled down, remove and streamline `pulseaudio` initialization so that there are less places for this to go wrong.
Differential Revision: https://phabricator.services.mozilla.com/D74809
This patch is adding an option to push a perftest run in the CI.
It's based on :
- sparse profiles
- push_to_try
- options passed through try_task_config.json
Differential Revision: https://phabricator.services.mozilla.com/D74115
Changes:
While the ubuntu1804 migration work was taking place, `pulseaudio` related pieces of code were sprinkled across a wide variety of modules.
Now that things have settled down, remove and streamline `pulseaudio` initialization so that there are less places for this to go wrong.
Differential Revision: https://phabricator.services.mozilla.com/D74809
Due to the resource usage (over 8Gb memory per libxul), cost and low impact, stop generating partials for asan builds
Differential Revision: https://phabricator.services.mozilla.com/D74710
|./mach try| subcommands are now compatible with both python 2 and 3.
Hand-tested with many combinations of subcommand and subcommand flags.
Updates tryselect unit tests to use Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D73398
Since we now run those tests in baremetal (which runs more parallel jobs
than packet.net), the variance of the time it takes to execute xpcshell
tests increased.
Differential Revision: https://phabricator.services.mozilla.com/D74703
In the 'chunk_by_runtime' algorithm a 'get_manifest' helper function is used to
determine what "manifest" a test belongs to. The logic is basically, if
'ancestor-manifest' exists then use that. Otherwise use 'manifest_relpath'.
We need to do this because in some cases a "shared" manifest can be included
multiple times from parent manifests, each with a different configuration.
However, when we calculate the "skipped" manifests in chunking.py, we were
simply using 'manifest_relpath' and ignoring 'ancestor-manifest'. I believe
this meant we were mis-reporting which manifests were skipped in the task logs.
It possibly even meant we were double-scheduling some tests (i.e, if the
'skip-if' was in the ancestor, not the shared one). I'm not sure if the double
scheduling was actually happening in practice, but it's certainly theoretically
possible.
Afaict, this only affected a handful of xpcshell manifests on Windows and
Android.
Depends on D74449
Differential Revision: https://phabricator.services.mozilla.com/D74451
This is instead of tests and will make it easier to re-use in the taskgraph.
This commit is a straight refactor and results in zero differences in the
taskgraph.
Depends on D74448
Differential Revision: https://phabricator.services.mozilla.com/D74449
When handling bug 1632429, I found some tests that worked on Python 2, but not Python 3.
They were marked accordingly as "expected failures". However, my system version of Python
is 3.8, while CI (and a non-trivial number of devs, probably) use 3.6.
Some of these tests marked as xfail were actually still working on versions of Python until 3.8.
The failure of this test was due to a change in default tarfile format. Explicitly setting this
format makes the tests pass in all relevant python versions.
Differential Revision: https://phabricator.services.mozilla.com/D74337
As |./mach| commands are migrated to python3, "taskgraph" needs to be compatible while still working with python2.
This patch migrates several iter*() calls and python2-specific imports to work with "six" instead.
Note that there's still python2-specific parts of taskgraph, I'm just modifying the pieces that are affecting
the code paths that I'm currently migrating (in this case, |./mach try|
Differential Revision: https://phabricator.services.mozilla.com/D73397
This patch changes the build variant we use for performance testing in raptor and browsertime from performance-test to the fennec-nightly build which is more representative of what a user might experience.
Depends on D68190
Differential Revision: https://phabricator.services.mozilla.com/D72886
This patch does a few things:
(1) Prevents Amazon and Youtube Fenix Browsertime tests from running on each push.
(2) Prevents Chrome Browsertime tests from running on each push (these should only run through the cron task).
(3) Prevents Speedometer Fenix Browsertime tests from running on each m-c push.
(4) Replaces the Raptor Fenix speedometer test in the general cron task with the Browsertime variant.
(5) Replaces the Raptor Fenix Amazon test with in the tp6m cron task with the Browsertime variant.
Differential Revision: https://phabricator.services.mozilla.com/D68190
This gets rid for the need of a number of local packages (mostly related
to Gtk+3). One exception is that we now need a 32-bits version of the
xz-utils package, some -dev package depends on it, and that dependency
can't be fulfilled in the 32-bits image because we already have the
64-bits backport installed, which conflicts with it (we need both
32-bits and 64-bits package to be at the same version when installed).
The system binutils fails to link clang-7 for some reason, so we now use
our toolchain binutils instead, like we already do for newer versions of
clang.
The debian-packages docker image now needs an explicit installation of
git, because it's not pulled in via the recommends of some other
package.
For some reason, snapshot.debian.org doesn't contain the jessie-backports
archive at the same location as others, and only has a few snapshots of
the archive.
Differential Revision: https://phabricator.services.mozilla.com/D73784
Now that upstream winchecksec builds and works natively on Linux, use
that. That should solve the random crashes under Wine. If random crashes
still happen, it will be easier to debug anyways.
We bump to the last version that doesn't use vcpkg because vcpkg makes
things more difficult.
Differential Revision: https://phabricator.services.mozilla.com/D73405
This is very similar to wcosta's earlier patch, with a few adjustments to overcome
special cases:
- remove the wrench override of adb path; handle in config instead
- fix taskcluster config syntax in a couple of places
- add android-sdk-linux fetch for test-verify
Differential Revision: https://phabricator.services.mozilla.com/D73157
This gets rid for the need of a number of local packages (mostly related
to Gtk+3). One exception is that we now need a 32-bits version of the
xz-utils package, some -dev package depends on it, and that dependency
can't be fulfilled in the 32-bits image because we already have the
64-bits backport installed, which conflicts with it (we need both
32-bits and 64-bits package to be at the same version when installed).
The system binutils fails to link clang-7 for some reason, so we now use
our toolchain binutils instead, like we already do for newer versions of
clang.
The debian-packages docker image now needs an explicit installation of
git, because it's not pulled in via the recommends of some other
package.
For some reason, snapshot.debian.org doesn't contain the jessie-backports
archive at the same location as others, and only has a few snapshots of
the archive.
Differential Revision: https://phabricator.services.mozilla.com/D73784
This gets rid for the need of a number of local packages (mostly related
to Gtk+3). One exception is that we now need a 32-bits version of the
xz-utils package, some -dev package depends on it, and that dependency
can't be fulfilled in the 32-bits image because we already have the
64-bits backport installed, which conflicts with it (we need both
32-bits and 64-bits package to be at the same version when installed).
The system binutils fails to link clang-7 for some reason, so we now use
our toolchain binutils instead, like we already do for newer versions of
clang.
The debian-packages docker image now needs an explicit installation of
git, because it's not pulled in via the recommends of some other
package.
For some reason, snapshot.debian.org doesn't contain the jessie-backports
archive at the same location as others, and only has a few snapshots of
the archive.
Differential Revision: https://phabricator.services.mozilla.com/D73784
While we're going to rebuild all the Debian-based docker images, we
might as well add one corrective thing that is going to be required
later on, and would require another rebuild, while doing it now would
skip that.
On Debian 10, the python3 distutils module is in a separate package,
python3-distutils, which didn't exist in previous Debian releases,
because back then the distutils module was part of the python3 package.
Fortunately, the python3-distutils-extra package existed on all the
Debian versions we use and depends on the distutils module, so it is a
convenient way to have the python3-distutils package pulled in without
altering the list of packages to install based on the Debian version.
Differential Revision: https://phabricator.services.mozilla.com/D73785
This gets rid for the need of a number of local packages (mostly related
to Gtk+3). One exception is that we now need a 32-bits version of the
xz-utils package, some -dev package depends on it, and that dependency
can't be fulfilled in the 32-bits image because we already have the
64-bits backport installed, which conflicts with it (we need both
32-bits and 64-bits package to be at the same version when installed).
The system binutils fails to link clang-7 for some reason, so we now use
our toolchain binutils instead, like we already do for newer versions of
clang.
The debian-packages docker image now needs an explicit installation of
git, because it's not pulled in via the recommends of some other
package.
For some reason, snapshot.debian.org doesn't contain the jessie-backports
archive at the same location as others, and only has a few snapshots of
the archive.
Differential Revision: https://phabricator.services.mozilla.com/D73784
This includes scripts that involve `tup`, jobs that build `tup` in automation, `tup.configure`, and related infrastructure and documentation.
Differential Revision: https://phabricator.services.mozilla.com/D73921
Passing Cr.ERROR to an Error constructor is incorrect since it just sets the
message of the error to the integer value of the Cr.ERROR. Cr.ERRORs need to be
used as the second argument to Component.Exception to correctly construct an
Exception object with its result property set to the Cr.ERROR value.
This was done automatically by an expansion of the new
mozilla/no-throw-cr-literal eslint rule that will be introduced in the next
commit.
Differential Revision: https://phabricator.services.mozilla.com/D28075
This patch adds a new similarity metric that will allow us to determine when content changes occur in live site tests. It also enabled to recorded sites so we can get a comparison of the quality of the recording (and difference) between it and the live site. There are 2D and 3D variants of this score which capture different things. The 2D score only looks at the final frame, so it gives a measure of how consistent/similar the end state is for the test. The 3D variant is more comprehensive and captures how the page was rendered.
Differential Revision: https://phabricator.services.mozilla.com/D73450
This patch adds the extra options settings to the visual metrics perfherder data by passing it through the jobs.json file.
Differential Revision: https://phabricator.services.mozilla.com/D73277
The visual metrics tasks were not being picked up by the cron task. This patch fixes this by correctly parsing for vismet tasks in the general_perf_testing cron task.
Differential Revision: https://phabricator.services.mozilla.com/D73294
The data isn't used anymore, as it's encoded in taskcluster/ci/l10n-bump/kind.yml.
This doesn't affect mobile, but TB/SM could use a corresponding change.
Differential Revision: https://phabricator.services.mozilla.com/D73712
Now that upstream winchecksec builds and works natively on Linux, use
that. That should solve the random crashes under Wine. If random crashes
still happen, it will be easier to debug anyways.
We bump to the last version that doesn't use vcpkg because vcpkg makes
things more difficult.
Differential Revision: https://phabricator.services.mozilla.com/D73405
There are cases where --recurse-submodules breaks things (e.g. when
newer versions of the repository remove a submodule). So don't
recurse-submodules at all at clone or checkout time, but instead
initialize and update submodules after the checkout.
Also don't checkout at clone time, because it's redundant with the
checkout, and we only really trust the explicit checkout anyways, so
it's better to not checkout during the clone.
Differential Revision: https://phabricator.services.mozilla.com/D73353
We starting doing that because snapshot.debian.org would ban some AWS IP
ranges, and we would get random failures, but that's not the case
anymore. OTOH, when more "normal" errors happen, like when you change a
Dockerfile to add a package, and that package actually doesn't exist,
the image build is tried 5 times, with no chance it will succeed, and
treeherder doesn't link to the log because it's purple, so you need to
manually go to taskcluster.
Removing the autoretry will make things smoother.
Differential Revision: https://phabricator.services.mozilla.com/D73392
Changes:
Add several build types to the blacklist, so that for `try fuzzy/try chooser` users, these will not show unless `--full` is applied. For `try syntax` users, these will become non-schedulable.
Differential Revision: https://phabricator.services.mozilla.com/D72068
With dynamic-test-selection, we'll also need to query the bugbug service from
the transforms. Let's move the querying logic to a utility file to share it
more easily.
Differential Revision: https://phabricator.services.mozilla.com/D73088
There were a fair number of changes, but the main ones are the switch to
a faster hash (blake3), and to tokio 0.2's faster scheduler.
Differential Revision: https://phabricator.services.mozilla.com/D72960
We use the term 'tests' to refer to 'tasks' in the tests.py transforms. Imo,
this makes things very hard to follow as the term 'test' is also used for all
kinds of other contexts in that file. Let's just call them what they are:
tasks.
I decided to land this as part of this series as I will be adding further uses
of the word 'test' later on.
Differential Revision: https://phabricator.services.mozilla.com/D73063
There were a fair number of changes, but the main ones are the switch to
a faster hash (blake3), and to tokio 0.2's faster scheduler.
Differential Revision: https://phabricator.services.mozilla.com/D72960
This adds a multi-e10s variant for geckoview-junit tests. With bug 1622944
resolved, the test suite passes, so we allow this variant to be tier-1.
Differential Revision: https://phabricator.services.mozilla.com/D66676
- changed test URL to match the dev server
- changed output.py in several places to fix new test names, dict keys, to cover all tests
- added amazonaws.com to manifest.json file to fix the loading issue for benchmark.js file
- added all raptor tests
- changed the constants for measure and alert_on
Differential Revision: https://phabricator.services.mozilla.com/D62546
The default way to split the 'arg' parameter for CompositeStrategies, is to
duplicate it across all substrategies. By setting 'split_arg=tuple', we instead
break the arg up so the first index goes to the first substrategy, the second
index goes to the second substrategy, etc.
This means that the length of the 'test' arg must be at least as long as the
number of substrategies.
Differential Revision: https://phabricator.services.mozilla.com/D72464
This patch prevents speedometer tests from running on all android builds and also prevents chrome tests from running there.
Differential Revision: https://phabricator.services.mozilla.com/D72727
This patch changes how Google Chrome for Android is deployed. Rather than relying on automatic updates, we will use tooltool to download the APK and install it ourselves. Some changes were done in taskcluster to remove a hack that was put in place to disable interal tooltool downloads (the issue is resolved now).
A tooltool manifest is added for this, and to keep ourselves organized, all manifests (including the playback ones) are moved into a folder called `tooltool-manifests`.
Differential Revision: https://phabricator.services.mozilla.com/D72198
