Quick fix for python3 mozbase perma-fail on osx: Use python 3.6 explicitly, rather
than the system default 3.7, which appears to be broken currently (lacking ssl support).
Differential Revision: https://phabricator.services.mozilla.com/D26345
--HG--
extra : moz-landing-system : lando
Ship-it v1 is going away soon and we won't need to create new releases in Ship-it v1 in parallel with Ship-it v2. It's time to prep patches to remove this functionality.
Differential Revision: https://phabricator.services.mozilla.com/D26044
--HG--
extra : moz-landing-system : lando
This imports the changes from wheezy-lts (http://deb.freexian.com/extended-lts/)
and creates a package we install in the debian7-based images (with a
modified version number to work around bug #1419577.
This leaves out debian7-raw and debian7-packages as unpatched, because
of the chicken-and-egg problem.
Depends on D26100
Differential Revision: https://phabricator.services.mozilla.com/D26102
--HG--
extra : moz-landing-system : lando
When docker images use setup_packages.sh, they add apt sources. While we
currently do run apt-get update to pick those new sources, if a package
provided by them is already installed and not explicitly listed in
subsequent apt-get install, they're not going to be upgraded.
Differential Revision: https://phabricator.services.mozilla.com/D26100
--HG--
extra : moz-landing-system : lando
We only run the cron job on release branches, so it will only get scheduled
there. By not otherwise restricting the job, it makes it easier to test the
cron job on other branches (like try).
Differential Revision: https://phabricator.services.mozilla.com/D26200
--HG--
extra : moz-landing-system : lando
This makes sure that the mozharness scripts have access to all the packages in
the build system virtualenv (namely mozbase).
Differential Revision: https://phabricator.services.mozilla.com/D22184
--HG--
extra : moz-landing-system : lando
The previous changes only changed the check in the release process. This also
updates the periodic check.
Differential Revision: https://phabricator.services.mozilla.com/D26199
--HG--
extra : moz-landing-system : lando
Now that we have added the necessary scopes to `ci-configuration`,
we can add the in-tree scopes to give tasks access to the
`hgmointernal` config Taskcluster secret.
Differential Revision: https://phabricator.services.mozilla.com/D25001
--HG--
extra : moz-landing-system : lando
This task runs on wpt metadata changes and uploads an artifact
containing the summarised metadata.
Depends on D24178
Depends on D24178
Differential Revision: https://phabricator.services.mozilla.com/D24179
--HG--
extra : moz-landing-system : lando
Avoid intermittent timeouts: increase max-run-time to 2.5 hours to account for
normal variability in run time.
Differential Revision: https://phabricator.services.mozilla.com/D27017
--HG--
extra : moz-landing-system : lando
I've refactored the artifact map generation slightly to make the list
of platforms more flexible, and also to let us have the previous name for
win64_aarch64_info.txt.
Differential Revision: https://phabricator.services.mozilla.com/D27049
--HG--
extra : moz-landing-system : lando
This makes sure that the mozharness scripts have access to all the packages in
the build system virtualenv (namely mozbase).
Differential Revision: https://phabricator.services.mozilla.com/D22184
--HG--
extra : moz-landing-system : lando
With the shift to shippable builds we no longer run tests on osx/opt though many still push to try with old try syntax using -p macosx64 and get surprised by no tests. This patch fixes it as a bandaid by appending macosx64-shippable as a platform when macosx64 is specified, making the tests run in the appropriate cases. The expectation with the methodology of this patch is that we'll be killing try syntax support in the near future, eliminating the need for these sorts of bandaids
Differential Revision: https://phabricator.services.mozilla.com/D26048
--HG--
extra : moz-landing-system : lando
There was special case logic to map the win64 platform to win32, for stub
entries. When win64-aarch64 was added no special case was added for that
plaform, so they stub entries pointed at the incorrect place. This changes the
configuration so that all stub entries point at the win32 paths, without
needing special case code.
Differential Revision: https://phabricator.services.mozilla.com/D25841
--HG--
extra : moz-landing-system : lando
We check that partials as a safety check. But we don't need that for staging builds,
and it is often useful to be able to test things that don't depend on partials.
The shipit UI currently still requires partials, but that can be worked around using
the react dev tools.
Differential Revision: https://phabricator.services.mozilla.com/D24834
--HG--
extra : moz-landing-system : lando
The original code was converting to json, then comparing against `{}`. This switches things
around so that json is only generated where it is directly used a json.
Differential Revision: https://phabricator.services.mozilla.com/D24833
--HG--
extra : moz-landing-system : lando
Enables the list of suites as defined in Bug 1540213.
- added new item in `test-sets.yml` for windows10-aarch64.
- reference the new test-set in `test-platforms.yml`.
Differential Revision: https://phabricator.services.mozilla.com/D25979
--HG--
extra : moz-landing-system : lando
These will run on android-hw against the latest reference browser nightly.
Since they are try-only, they can only be scheduled with |mach try fuzzy
--full|.
Differential Revision: https://phabricator.services.mozilla.com/D25801
--HG--
extra : moz-landing-system : lando
This is needed for cross-language LTO (bug 1512723). We don't want to block on waiting for 1.34's release, so we'll get a head start now, but we'll update to the final 1.34 release when available. Rust Forge estimates the release at 11 April.
Differential Revision: https://phabricator.services.mozilla.com/D25851
--HG--
extra : moz-landing-system : lando
this change adds support for parallel gcp builds for the following windows
build configurations:
- win32
- opt
- debug
- pgo
- win64
- opt
- debug
- pgo
gcp builds are triggered with a treeherder tier 3 flag so that they are only
displayed in the treeherder ui when the user has a tier 3 flag set.
gcp builds use a th build symbol of "Bg" to make them easy to differentiate
from ec2 builds in the treeherder ui.
gcp builds use a perfherder "gcp" flag to make them easier to differentiate
from ec2 builds in the perfherder ui.
Differential Revision: https://phabricator.services.mozilla.com/D24865
--HG--
extra : moz-landing-system : lando
We weren't including this before, which was causing us to not track size
metrics for nss.dll and xul.dll, which is suboptimal.
Differential Revision: https://phabricator.services.mozilla.com/D25792
--HG--
extra : moz-landing-system : lando
This patch introduces a new marionette media test along
with a Youtube test.
To run the Youtube streaming test locally:
./mach marionette-test dom/media/test/marionette/test_youtube.py -vv --gecko-log -
Differential Revision: https://phabricator.services.mozilla.com/D23644
--HG--
extra : moz-landing-system : lando
This patch introduces a new marionette media test along
with a Youtube test.
To run the Youtube streaming test locally:
./mach marionette-test dom/media/test/marionette/test_youtube.py -vv --gecko-log -
Differential Revision: https://phabricator.services.mozilla.com/D23644
--HG--
extra : moz-landing-system : lando
An `sy-tp6` variant is added to the AWSY test suite that runs against the tp6 pageset.
Differential Revision: https://phabricator.services.mozilla.com/D24585
--HG--
extra : moz-landing-system : lando
This also renames win_taskcluster_unittest.py to win_unittest.py for
consistency with the other platforms.
Differential Revision: https://phabricator.services.mozilla.com/D25401
--HG--
extra : moz-landing-system : lando
We don't actually use the build-tools repo in-tree anymore, so remove the
support code for it.
Differential Revision: https://phabricator.services.mozilla.com/D25631
--HG--
extra : moz-landing-system : lando
We apply a local patch while we wait for upstream wine and mingw to review
the changes to widl that are necessary to generate the correct headers. Here we
just grab the generated headers and patch them into MinGW
We can revert this when MinGW updates, but for now we would like to unblock
the ANGLE update
Depends on D25294
Differential Revision: https://phabricator.services.mozilla.com/D25295
--HG--
extra : moz-landing-system : lando
This is needed to bring dispatcherqueue.h in, which is needed for
an ANGLE upgrade. It also ensures that overloads for secure string
functions are always defined and removes redundant --enable-secure-api
configure option and use of MINGW_HAS_SECURE_API
Differential Revision: https://phabricator.services.mozilla.com/D25294
--HG--
extra : moz-landing-system : lando
Run DocUp as tier 2 rather than tier 3, to make the task visible by default
and get the benefit of at least some sheriffing.
Differential Revision: https://phabricator.services.mozilla.com/D25319
--HG--
extra : moz-landing-system : lando
This sets things enough things up to be able to push to try with an
opt-in, but doesn't run the job on every push. This can be used as a
template for future work on a fuzzing job.
Differential Revision: https://phabricator.services.mozilla.com/D25069
--HG--
extra : moz-landing-system : lando
This only uses cross-platform tools, so switch to running these on linux, which
cuts the runtime down from ~20m to ~3m.
Differential Revision: https://phabricator.services.mozilla.com/D1080
--HG--
extra : moz-landing-system : lando
This also switches it to use the generic toolchain build image, as
it is no longer being used exclusively by mingw builds.
Differential Revision: https://phabricator.services.mozilla.com/D24230
--HG--
extra : moz-landing-system : lando
This avoids the surprising result of `try_task_config.json` overriding explicit
`--target-tasks-method` from a cron task run against a try push.
Differential Revision: https://phabricator.services.mozilla.com/D25089
--HG--
extra : moz-landing-system : lando
This runs crashtests and reftests on ASAN builds with WR enabled, so
that we catch any ASAN regressions prior to landing without incurring
too great of a test load hit.
Differential Revision: https://phabricator.services.mozilla.com/D24952
--HG--
extra : moz-landing-system : lando
The updated cargo-apk version now correctly handles the `--frozen` flag
and additionally translates it to the `--offline` flag when invoking
gradle. This makes the gradle build fail instead of attempting network
fetches. To make the offline gradle build work, we set up a build.gradle
snippet that points to the maven repositories from the gradle toolchain
artifact, and have cargo-apk use that instead of the default jcenter()
repository.
Differential Revision: https://phabricator.services.mozilla.com/D24485
--HG--
extra : moz-landing-system : lando
No functional change here. This just extracts the existing script into a
helper file and shifts things around slightly so it's more logically
grouped (the env variables are needed for the cargo-apk invocation).
Also use better bash hygiene with variables.
Differential Revision: https://phabricator.services.mozilla.com/D24484
--HG--
extra : moz-landing-system : lando
The original globs were only finding C++ files in the root of the repository.
Differential Revision: https://phabricator.services.mozilla.com/D25046
--HG--
extra : moz-landing-system : lando
Run gv-docs as tier 2 rather than tier 3, to make the task visible by default
and get the benefit of at least some sheriffing.
Differential Revision: https://phabricator.services.mozilla.com/D24956
--HG--
extra : moz-landing-system : lando
Makes most kinds that reference nightly attribute reference the shippable attribute.
Also makes most transforms that use nightly use shippable
Transfers dependencies/ownership for some things to shippable from nightly where it was harder to support both.
In no particular order, full list:
Send shippable attribute down to dep tasks.
Set tests as shippable attribute
Add new signing routes
Add shippable routes to repackage_routes transform
Adjust target tasks
Add shippable nightly-l10n
Add nightly-signing and as a side affect add repackage and repackage-signing
Add support for proper balrog platforms for shippable
Add shippable to the nightly sccache guard
Fix TC_PLATFORM_PER_FTP in partners.py to use shippable
Add shippable to mozharness_test variants
Only actually used for android which doesn't have shippable at this time.
Add shippable variant to beetmover transforms
Do nightly signing for mars on shippable
Support shippable in partner-repack transform
Support shippable in amo langpacks transform
Use proper signing for shippable tasks in repackage transforms
Set upload symbols to use shippable too
Use shippable as deps for geckodriver extraction
Use shippable as dep for autograph-stage signing
Do not run beetmover-l10n for shippable
Run shippable style jobs for repackage signing
Set build_platform for update verify and uvc to be shippable
Run repackage-msi for shippable
Add shippable to osx partner repack signing
add shippable to emefree repackage
add shippable to emefree repackage signing
add shippable to beetmover checksums
Add shippable to partner repack repackage signing
add partner repack beetmover
Add shippable to mar signing
Add shippable to mar-signing-l10n
add shippable to eme free beetmover checksums
Add shippable to upload-generated-sources
Add beetmover langpacks to shippable
Add repackage-l10n to shippable
Add shippable to partner repack chunk-dummy
Do eme free builds with shippable
Add signing of language packs to shippable
Add snap repackage for shippable
Add shippable for release-eme-free repack signing
Add partials for shippable
Add partner repack repackage for shippable
Add emefree beetmover for shippable
Add checksums-signing to shippable
Switch partner repacks to shippable
Add shippable to beetmover-repackage
Add secondary update verify configs for shippable
secondary update verify for shippable
Differential Revision: https://phabricator.services.mozilla.com/D24699
--HG--
extra : moz-landing-system : lando
Jmaher indicated we do not have the test capacity to incur this as a duplicated set
Differential Revision: https://phabricator.services.mozilla.com/D23382
--HG--
extra : moz-landing-system : lando
This also relates to Bug 1522111 where we turned off opt tests in favor of pgo,
shippable is like the new pgo so do that.
This has a side affect of adding talos-tp6-stylo-threads to inbound for
osx-shippable (previously was only on autoland).
This has no practical affect after D23382 is applied (because of lack of OSX capacity)
Differential Revision: https://phabricator.services.mozilla.com/D23129
--HG--
extra : moz-landing-system : lando
Should there end up being a need we can back out this patch and let them run, but :jmaher indicated
he was happy with dropping them entirely and not duplicating.
Differential Revision: https://phabricator.services.mozilla.com/D23383
--HG--
extra : moz-landing-system : lando
This is useful in order to not have to run linux64/opt on push, especially on autoland/inbound
when we need a source test. It is also required if we remove the linux64-pgo build type entirely.
Differential Revision: https://phabricator.services.mozilla.com/D23124
--HG--
extra : moz-landing-system : lando
I did a bunch of manual testing with this, the biggest uncertainties lie around beta and central/nightly.
We are adding shippable-qr to beta because of replacing nightly too.
Autoland and inbound should have the same sets of tasks.
beta -
- adds mochitest-plain-headless-{1..4} to beta (not currently run)
- adds raptor to run on shippable for beta
- currently runs on opt on beta, and for nightly tasks on beta only webaudio-chrome runs.
- adds talos to shippable tasks, on beta talos only runs against opt.
central -
- adds browser-screenshots to nightly graph
- adds mochitest-plain-headless-{1..4} to nightly graph
- adds browser-instrumentation to shippable
Differential Revision: https://phabricator.services.mozilla.com/D23122
--HG--
extra : moz-landing-system : lando
Effectively back out much of the run on projects from D22710
This also has the added affect of scheduling the shippable builds to run because of dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D22833
--HG--
extra : moz-landing-system : lando
raptor-chrome is Google Chrome and only needs to run once per day, so mozilla-central pushes and try.
raptor-profiling is primarily for devs to have up to date profile information and it too only needs to run once per day.
TODO is to try and find a clean way to make them only run when we trigger Nightlies rather than every m-c push.
Differential Revision: https://phabricator.services.mozilla.com/D22832
--HG--
extra : moz-landing-system : lando
Makes most kinds that reference nightly attribute reference the shippable attribute.
Also makes most transforms that use nightly use shippable
Transfers dependencies/ownership for some things to shippable from nightly where it was harder to support both.
In no particular order, full list:
Send shippable attribute down to dep tasks.
Set tests as shippable attribute
Add new signing routes
Add shippable routes to repackage_routes transform
Adjust target tasks
Add shippable nightly-l10n
Add nightly-signing and as a side affect add repackage and repackage-signing
Add support for proper balrog platforms for shippable
Add shippable to the nightly sccache guard
Fix TC_PLATFORM_PER_FTP in partners.py to use shippable
Add shippable to mozharness_test variants
Only actually used for android which doesn't have shippable at this time.
Add shippable variant to beetmover transforms
Do nightly signing for mars on shippable
Support shippable in partner-repack transform
Support shippable in amo langpacks transform
Use proper signing for shippable tasks in repackage transforms
Set upload symbols to use shippable too
Use shippable as deps for geckodriver extraction
Use shippable as dep for autograph-stage signing
Do not run beetmover-l10n for shippable
Run shippable style jobs for repackage signing
Set build_platform for update verify and uvc to be shippable
Run repackage-msi for shippable
Add shippable to osx partner repack signing
add shippable to emefree repackage
add shippable to emefree repackage signing
add shippable to beetmover checksums
Add shippable to partner repack repackage signing
add partner repack beetmover
Add shippable to mar signing
Add shippable to mar-signing-l10n
add shippable to eme free beetmover checksums
Add shippable to upload-generated-sources
Add beetmover langpacks to shippable
Add repackage-l10n to shippable
Add shippable to partner repack chunk-dummy
Do eme free builds with shippable
Add signing of language packs to shippable
Add snap repackage for shippable
Add shippable for release-eme-free repack signing
Add partials for shippable
Add partner repack repackage for shippable
Add emefree beetmover for shippable
Add checksums-signing to shippable
Switch partner repacks to shippable
Add shippable to beetmover-repackage
Add secondary update verify configs for shippable
secondary update verify for shippable
Differential Revision: https://phabricator.services.mozilla.com/D24699
--HG--
extra : moz-landing-system : lando
Jmaher indicated we do not have the test capacity to incur this as a duplicated set
Differential Revision: https://phabricator.services.mozilla.com/D23382
--HG--
extra : moz-landing-system : lando
This also relates to Bug 1522111 where we turned off opt tests in favor of pgo,
shippable is like the new pgo so do that.
This has a side affect of adding talos-tp6-stylo-threads to inbound for
osx-shippable (previously was only on autoland).
This has no practical affect after D23382 is applied (because of lack of OSX capacity)
Differential Revision: https://phabricator.services.mozilla.com/D23129
--HG--
extra : moz-landing-system : lando
Should there end up being a need we can back out this patch and let them run, but :jmaher indicated
he was happy with dropping them entirely and not duplicating.
Differential Revision: https://phabricator.services.mozilla.com/D23383
--HG--
extra : moz-landing-system : lando
This is useful in order to not have to run linux64/opt on push, especially on autoland/inbound
when we need a source test. It is also required if we remove the linux64-pgo build type entirely.
Differential Revision: https://phabricator.services.mozilla.com/D23124
--HG--
extra : moz-landing-system : lando
I did a bunch of manual testing with this, the biggest uncertainties lie around beta and central/nightly.
We are adding shippable-qr to beta because of replacing nightly too.
Autoland and inbound should have the same sets of tasks.
beta -
- adds mochitest-plain-headless-{1..4} to beta (not currently run)
- adds raptor to run on shippable for beta
- currently runs on opt on beta, and for nightly tasks on beta only webaudio-chrome runs.
- adds talos to shippable tasks, on beta talos only runs against opt.
central -
- adds browser-screenshots to nightly graph
- adds mochitest-plain-headless-{1..4} to nightly graph
- adds browser-instrumentation to shippable
Differential Revision: https://phabricator.services.mozilla.com/D23122
--HG--
extra : moz-landing-system : lando
Effectively back out much of the run on projects from D22710
This also has the added affect of scheduling the shippable builds to run because of dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D22833
--HG--
extra : moz-landing-system : lando
raptor-chrome is Google Chrome and only needs to run once per day, so mozilla-central pushes and try.
raptor-profiling is primarily for devs to have up to date profile information and it too only needs to run once per day.
TODO is to try and find a clean way to make them only run when we trigger Nightlies rather than every m-c push.
Differential Revision: https://phabricator.services.mozilla.com/D22832
--HG--
extra : moz-landing-system : lando
This task runs on wpt metadata changes and uploads an artifact
containing the summarised metadata.
Depends on D24178
Differential Revision: https://phabricator.services.mozilla.com/D24179
--HG--
extra : moz-landing-system : lando
Taskcluster has authorative information about the repository being built, so
pass that to mozharness, rather than have mozharness reconstruct it from
hand-maintained mozharness config.
Differential Revision: https://phabricator.services.mozilla.com/D24561
--HG--
extra : moz-landing-system : lando
Bug 1534522 added win64-aarch64-eme/opt builds, which are artifact builds
that glue together a win64-aarch64/opt build and a win32/opt build.
This enables EME on the corresponding nightlies in a slightly different
way:
- this adds a no-eme build that corresponds to win64-aarch64/opt.
- this turns the existing nightly into an artifact build that glues
together that no-eme build and the win32 nightly.
The no-eme build cannot have the nightly attribute set, first because
the beetmover transform fails in that case, and because that would imply
shipping those builds, but they're not meant to be shipped this way.
It also has run-on-projects set to an empty list so that it doesn't
appear by default in `mach try fuzzy`, while still being triggered when
needed due to being a dependency of the nightly build.
It is preferable to keep the win64-aarch64{,-eme}/opt builds untouched
to make things easier for try (the win64-aarch64 ones being the main
ones to try; also, the -eme builds currently fail with --artifacts).
Ideally, like in bug 1534522, we'd add a diffoscope build to ensure
the variations between the nightly and its base no-eme build are within
control, but currently, that would trigger nightlies on every push,
which is not desirable. Ideally, they'd trigger whenever both their
dependencies are in the target task graph. We leave that to a followup.
Differential Revision: https://phabricator.services.mozilla.com/D23640
The code that actually downloads it is behind a condition that isn't set
anywhere.
Differential Revision: https://phabricator.services.mozilla.com/D24215
--HG--
extra : moz-landing-system : lando
Currently, if the files match and you try to look at the diff, you get
{
"reason": "file-missing-on-worker",
"message": "Artifact \"public/diff.html\" not found at \"/builds/worker/diff.html\""
}
which makes it hard to tell if there was an error generating a diff, or if the files matched.
This changes things to ask diffoscope to always output a diff, to remedy that confusion.
Differential Revision: https://phabricator.services.mozilla.com/D24316
--HG--
extra : moz-landing-system : lando
Searchfox relies on the bugzilla component job running on the same push
as the indexing jobs, but there's nothing that actually guarantees that.
Thus far pushes to m-c pretty much always have source changes so the
bugzilla component job gets run, but on beta/release branches it's
possible to get pushes with just tag changes and no source changes, so
the bugzilla component job would get optimized away. This patch ensures
that the job gets run along with the other indexing jobs that searchfox
needs.
Differential Revision: https://phabricator.services.mozilla.com/D24507
--HG--
extra : moz-landing-system : lando
Without this the shippable builds take >20 seconds to compute the prune through new_as_old_is_high_value, locally. With this it is near instant.
Differential Revision: https://phabricator.services.mozilla.com/D22711
--HG--
extra : moz-landing-system : lando
This sets all of the shippable tests to not run in the places where they would otherwise.
This patch will be effectively undone later in the patchset.
Differential Revision: https://phabricator.services.mozilla.com/D22710
--HG--
extra : moz-landing-system : lando
When we set the nightly attribute the tasks don't run on-push, so we use a new attribute.
Differential Revision: https://phabricator.services.mozilla.com/D22709
--HG--
extra : moz-landing-system : lando
