The Windows code coverage build is using LLVM, so we need to tell grcov that the artifacts are in the LLVM format.
--HG--
extra : rebase_source : 8507ce4e3614b24275350e48a761fd8a129e0e39
It seems psutil can throw a wide range of exceptions when accessing system
information on aws...intermittently, of course. Let's expect and discard
such exceptions so that test jobs are not dependent on creating system-info.log.
According to sfink, mozharness is no longer used to drive hazard
builds. That means a lot of dead code that can be removed.
After this commit, there are no more references to "hazard" or
"spidermonkey" in testing/mozharness.
MozReview-Commit-ID: 8MWl8dMwRTD
--HG--
extra : rebase_source : 2156fbd13dffb22bb08b10fec2a66a9eebde8d57
These scripts are included by hazard-analysis.sh. That's their only
reference in repo.
We could probably inline these scripts. But let's start by moving them
out of mozharness since no active mozharness based task is using them.
MozReview-Commit-ID: 13oen42Txmh
--HG--
rename : testing/mozharness/scripts/spidermonkey/build.browser => taskcluster/scripts/builder/hazard-browser.sh
rename : testing/mozharness/scripts/spidermonkey/build.shell => taskcluster/scripts/builder/hazard-shell.sh
extra : rebase_source : 782f7b3f3537cfefb51b0e5f1b459c8ad0daca5b
I'm seeing "try" in my repacks, when the underlying build has
"nightly-try". This should make the two agree.
MozReview-Commit-ID: 45yE9Qwz0v7
--HG--
extra : rebase_source : ff1ae4e50203ea032032069203558d75d348ff21
Single-locale repacks need to run aapt (--without-gradle) or Gradle
(--with-gradle). When running --with-gradle, they need to compile the
Java source code again (in order to produce a fresh R.java with
correct IDs). That compile will be part of the shipping APK, so it
needs to be configured "the same" as the underlying repacked. *This
is a significant change in behaviour, but necessary to support newer
Gradle/aapt versions, which do not maintain R.java ID mappings across
invocations.*
Part of the configuration are the secret keys and features that are
gated on them. This commit makes those secrets available to
single-locale repacks.
MozReview-Commit-ID: 4REPsIb5TgN
--HG--
extra : rebase_source : 2d23e0e0c51a61e50acf24123b316bdbb0b579ff
extra : source : a721890f7573140ca6a926e722bd3538c732dae7
I'm seeing "try" in my repacks, when the underlying build has
"nightly-try". This should make the two agree.
MozReview-Commit-ID: 45yE9Qwz0v7
--HG--
extra : rebase_source : 8b524470680e248c649bf3e4e532cdd5487ec538
Single-locale repacks need to run aapt (--without-gradle) or Gradle
(--with-gradle). When running --with-gradle, they need to compile the
Java source code again (in order to produce a fresh R.java with
correct IDs). That compile will be part of the shipping APK, so it
needs to be configured "the same" as the underlying repacked. *This
is a significant change in behaviour, but necessary to support newer
Gradle/aapt versions, which do not maintain R.java ID mappings across
invocations.*
Part of the configuration are the secret keys and features that are
gated on them. This commit makes those secrets available to
single-locale repacks.
MozReview-Commit-ID: 4REPsIb5TgN
--HG--
extra : rebase_source : 9a74ea5f6633d1a4bd52d6116b60edaf974d11eb
extra : source : a721890f7573140ca6a926e722bd3538c732dae7
This also turns the tier 2 job B(n)g into tier 1, since moz.build is
still tier 1. It also pushes a lot of GeckoView related tasks into
the main builds, since they should run as part of Gradle builds.
This also removes unused tooltool manifests; the jobs that used these
manifests use only toolchain tasks now.
MozReview-Commit-ID: 2GmnJ7joCTT
--HG--
extra : rebase_source : 75cd2dfb51e0e1b510f5e618c2dc881cf5f22bf2
extra : source : 6b95b09d6afbb83ba89c47b237dfce6e15587bbe
This should be functionally identical to `make -f client.mk`.
MozReview-Commit-ID: GfKHtezLBNL
--HG--
extra : rebase_source : e7965718b16d22055abdab2838e97cae63330ce9
The old code was simply running configure and manually invoking some
make targets via client.mk. These can both be done via `mach`.
As part of the change, the build targets have been consolidated. There
is still an abstraction leak here. But at least we aren't using client.mk.
MozReview-Commit-ID: 7oMXPWPZS6V
--HG--
extra : rebase_source : 6d632dc086d79a17e577da66336c77003d963f67
Building with "Linux x64 JSDCov", each suite of tests will produce a list of JSDCov artifacts (.json). This patch compresses all JSDCov artifacts into 1 zip file for each suite upon tests completion.
MozReview-Commit-ID: DaNLOQuW2lc
--HG--
extra : rebase_source : cceb912d0fe16f9921e1d13fe85d53af2abcb29c
This patch adds fennec staging and production relpro configs.
MozReview-Commit-ID: AuY21bqU6bl
--HG--
extra : rebase_source : 209c9d9971d2ee18a7c794e0350af90f05d6b705
In bug 1360587, we removed bouncer.apk. We accidentally removed the
mozharness config files for bouncer submission in d72b09c9228c as well.
This patch merely adds the files back.
MozReview-Commit-ID: 8c7lFr6Dpz1
--HG--
extra : rebase_source : 17ef6f2de0b0b6e1f8fde50ff3141e23cda160d3
Upgrade from optparse to argparse:
1. 'type' field now needs to be callable (deleted if type was 'string' as that is the default)
2. 'extend' action re-implemented for argparse
3. 'callback' action no longer exists, re-implemented as a custom argparse action (only used in buildbase.py)
4. minor api changes, e.g 'add_option' -> 'add_argument'
MozReview-Commit-ID: HcKowF13Da3
--HG--
extra : rebase_source : e5e8160d91263fb273f790dbda5e2c2b2e02eaf6
This version of the sdk download is based on the previous one but includes
a new version - 1.0.39 - of adb and eliminates some files not used by the
emulator (smaller, faster download). By including adb in the sdk download used
for the emulator, the tools sdk download can be eliminated.
It's unfortunate, but somehow the
run-task > mozharness > mach build > mach gradle > Gradle
causes Gradle to hang with no output. We avoid the bug by avoiding
the `max_build_output_timeout > 0` branch in mozharness. (These tasks
are all in Task Cluster, so they'll be killed eventually and it's not
a huge issue to remove the output timeout.)
MozReview-Commit-ID: 9wiLWS7kRQ9
--HG--
extra : rebase_source : 8c1ec73945494d3307148b3e7d0cd44815bab40a
This removes the option to pass in a custom virtualenv binary and gets
mozharness to always use the copy in 'external_tools'. Because this is now
deterministic, we don't need to do things like check what version of pip is
being used.
MozReview-Commit-ID: GWxkmaZVpsZ
--HG--
extra : rebase_source : 7b82fbf84fe272cf2adcbc1b01c9aa4b36e2af22
Bug 1382564 made the `mach artifact toolchain` invocations from
mozharness use the MOZ_TOOLCHAINS environment variable when it's set by
taskcluster through the decision task, so that toolchain dependencies
from the task graph are used, but the mozharness code is still skipping
mach artifact toolchain when MOZ_TOOLCHAINS is set but there is no
tooltool manifest set. Most jobs today still have a tooltool manifest
set, but jobs shouldn't need a dummy tooltool manifest to use toolchain
dependencies automatically.
--HG--
extra : rebase_source : 0437a8f3d43a83ffe32c4192f86ee9a621977e3e
We need mozharness configurations and mozconfigs for rusttests. We are
explicitly not doing Windows debug configurations currently because of
peculiar link errors in such configurations.
This patch:
- removes the obsolete mozilla-aurora l10n-bumper config.
- adds both central and beta format desktop bumper configs to jamun for testing.
- updates the central and beta configs to add desktop.
- updates the script to support the desktop configs.
We now support an `ignore_config` which acts like the `ignore-platforms` attribute.
MozReview-Commit-ID: KGwo0bRibw4
--HG--
extra : rebase_source : 1014c8d46104fc3b05586aa64f207cf38f37f98f
The main bug this fixes is that on reftest, the objdir needs to be added to the
whitelist on Windows. However, this only happens when running on Linux for some
reason.
Changing the --work-path and --obj-path arguments to --sandbox-read-whitelist
was more of a drive-by cleanup than anything necessary.
MozReview-Commit-ID: Dq8ZLETMzeM
--HG--
extra : rebase_source : 3d2cdda125205e76f86235eb373074899fe0789a
This is to avoid hitting https://github.com/pypa/pip/issues/510.
MozReview-Commit-ID: 7TK4DdbpKRD
--HG--
extra : rebase_source : ffef899d80864a81a81a1a332a9da1f949a05551
This fixes Single Locale Repacks for Fennec 57.0 betas.
This broke because our merge scripts rewrote l10n-central to mozilla-beta expecting the l10n repo to be called 'mozilla-beta', which is no longer true with cross-channel l10n. When we patched the configs we didn't see a 'mozilla-beta' entry here, butmissed that the merge-day scripts change it on us.
MozReview-Commit-ID: F7BJzpZg0Xj
--HG--
extra : source : cc652dcb13dba40ae3f263ae89ce3e610a34165f
extra : amend_source : 9fa72e4b2a662a6a63ddfc59dbacf7f03e57dd67
Complications:
- had to copy ReftestManifest into a test zip
- reftest harness was emitting multiple suite_start log entries with --repeat
- some extra path manipulation required to find reftests
This uploads a JSON summary of the results, without extra logging or
expectation data or anything. It is mostly useful for comparing the
results from two runs e.g. in a dashboard.
MozReview-Commit-ID: Ac45NVBxhy8
--HG--
extra : rebase_source : 2c7a82d6e46b8c9f7af8ad34559e51d84ba4e4ba
Currently only the Mochitests on Linux use a startup_timeout of 180s
for Marionette. This would also be necessary for the other platforms
(MacOS and Windows).
MozReview-Commit-ID: TGO7XDLXdI
--HG--
extra : rebase_source : 29219cc51404cebf22e894966294eae687bb5d68
This change rides the train along with the change to build beta from
l10n-central.
Not changing 52 esr config, as that continues to build from the
repository it's currently building from.
ESR will be picked up when the next ESR is riding the trains from central.
MozReview-Commit-ID: F9Wo2mcZ3PA
--HG--
extra : rebase_source : dbf6c39cc5ab51bf4970caa4d62a80060b6b8c7c
Notable changes
* ensure we run dump-symbols and upload actions on all platforms
* On android:
* add configuration and support for aarch64
* set min_sdk levels to match Fennec builds
* use a full copy of the r11c ndk (our truncated one was missing toolchains we needed) and set NDKROOT when calling build
* ensure the tooltool provided sdk is on the PATH
* on linux copy tooltool.py into the mock environment, so we can get dump_syms from tt
* remove macosx32 config as we've deprecated that in Firefox builds
* update dump_syms to recent m-c, notably for aarch64 support on linux
* on linux rev e365137fa61bfd729617ba1ebf9f1ed79facd1f2 (via try 0f72a5c28be1cdc2f3bdfaafdf3826254f6ba077)
* on mac rev e365137fa61bfd729617ba1ebf9f1ed79facd1f2 (via compile on a bld-lin-r5)
* on windows rev a4a448ba7f187069fce916ee234a06cbb0d06f80 (via try dc8b121e3c08e8022d62c0fa1951dd3dc4d6f7cc)
* switch to Visual Studio 2015 Update 3 on win32/win64 to match Firefox
* many updates to environement variables
* painful to get win64 right to run win32 dump_syms.exe, but that's why the x86 redist is on teh PATH
* unwind the changes to get_output_from_command() in v1.6 patch to avoid affecting other builds, and use query_env() which has this support already
* add a scp_upload_directory since we don't have rsync on windows, use that to talk to the ffxbld upload host (not a long term solution but OK for now)
Applies on top of https://reviewboard.mozilla.org/r/64022/diff/4#index_header
MozReview-Commit-ID: B3NiWFvr2oR
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
From version-control-tools revision
0451f128550d8c9dcf83d6628dea5a8030d5cc08. Vendored without
modifications.
This should be a rubber stamp review since the changes to
robustcheckout were already reviewed.
MozReview-Commit-ID: 3J0db7Cja1D
--HG--
extra : rebase_source : 4e36dc83a486841d253cd365ce7a6ceeb50a8ccc
On Taskcluster machines, /tools/buildbot doesn't exist. It turns out, nothing
refers to exes['python'] anyway, so we can just remove that (the preference is
to use sys.executable instead).
The references to exes['buildbot'] were all for sendchanges from builds to
tests, and those too are all gone now.
Lines like
'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
Were committing two sins: first, using a python executable from a random
virtualenv; and second, using a virtualenv.py from another random directory (in
this case, it's a utility script for a PuppetAgain module). Such lines are
replaced with a reference to /tools/virtualenv/bin/virtualenv, which is
installed for the express purpose of providing a virtualenv binary on testers
(for builds, we use the vendored copy).
MozReview-Commit-ID: 4iHX3B3MLyK
--HG--
extra : rebase_source : e4b4902a19d688b148c136bd10c706fc127cbf2d
The robustcheckout extension from revision
134574b64ddfa4d7c31977d792761cceca67665a of the version-control-tools
repo is vendored without modifications.
Changes since last time include printing of the Mercurial version and
more robust handling repositories not using modern storage
requirements.
This patch was not explicitly reviewed by glob. But glob reviewed all
the robustcheckout changes since the last vendor. So by the transitive
property of code review...
MozReview-Commit-ID: CejuVVGpaEy
--HG--
extra : rebase_source : d24dd19357c44f50f7605e974d91434bac3c47f8
extra : source : 68cdc2d1184dec80455ba0ea1ebbcab232b8c119
We use `hg init` to create the directory. Because this is what
typically occurs.
We also remove the disabling of "dotencode" in the hgrc. This was added
in https://hg.mozilla.org/build/mozharness/rev/b1dbc0f56ff8 (bug 857853)
for reasons that are unclear to me. We should never disable dotencode
because it may make some repositories not clonable on Windows
filesystems.
Disabling dotencode will also interfere with the latest version of
robustcheckout, which requires its presence.
MozReview-Commit-ID: 35qBsgwp0uW
--HG--
extra : rebase_source : b786fb38f6b09da9614ac40f5de8231b8305bf5d
We use Mercurial 4.3.1 pretty much everywhere in CI now.
Mozharness should be testing with it as well.
MozReview-Commit-ID: HT2rocEvdIe
--HG--
extra : rebase_source : 148a9cd82b18e693ee570f31fc961373e8466a3c
The robustcheckout extension from revision
134574b64ddfa4d7c31977d792761cceca67665a of the version-control-tools
repo is vendored without modifications.
Changes since last time include printing of the Mercurial version and
more robust handling repositories not using modern storage
requirements.
This patch was not explicitly reviewed by glob. But glob reviewed all
the robustcheckout changes since the last vendor. So by the transitive
property of code review...
--HG--
extra : rebase_source : f511876ecc743e6b453118297d327e9a07bb9441
History suggests these have been in place for a long time, back to an era when
we used split-horizon DNS and DNS service was flaky. We don't see this sort of
error very often anymore.
Now we have docker images that do not have nslookup installed, and this is
causing retries to fail.
The other option is to use `socket.getaddrinfo` here, but that introduces new
failure mode and could prevent retries that would otherwise succeed. The least
failure-prone approach is just to remove the calls.
MozReview-Commit-ID: JDjhYxUrROS
--HG--
extra : rebase_source : 2e29e9ca19343feea9c505622753cbac1d91012e
This will change all build symbols to 'Ba' and set the USE_ARTIFACTS=1 environment variable.
Mozharness will detect this env to decide whether to perform an artifact build or not.
MozReview-Commit-ID: J8HVZzOt4mX
--HG--
extra : rebase_source : 453028d9be5cb2ad07e9a2a8b769cb6aac9893fe
Other related changes:
* Only target.{zip,tar.gz,tar.bz} get the widevine signature (not the various installers).
* Linux{32,64}-nightly are now repackaged. Their mar files are not signed during signed step anymore. It now happens after the repackage.
* As a consequence, funsize routes for linux are now set to repackage-signing (instead of signing)
* Signed upstream artifacts are now defined in a dedicated module (to avoid duplication)
* Platforms defined in beetmover_repackage now allow regex (to reduce duplication too)
* Mozharness configs: Delete unused (and misleading) `src_mozconfig` for windows. This value is actually not used when `run_configure` (in the same dict) is set to False.
MozReview-Commit-ID: COKqevW9Mzn
--HG--
extra : source : ffc2e43aa834e05f0d51d68dfb36317c1b408b08
From revision 19db5f4b5b10f639d24e69a4f21e4e80c6e5bbdd of
version-control-tools. Previous revision was
249a47720ddcf896a9f07600c429a1b4492b805e.
Changes include:
* Use new vfs APIs when available (bug 1359959)
* Mark as compatible with Mercurial 4.2 (bug 1361182)
* Retry after SSLError (bug 1354824)
* Mark as compatible with Mercurial 4.3
* Detect and recover from open locks (bug 1297153)
The most significant is the last one. TaskCluster can SIGKILL
`hg`, leading to orphaned transactions and locks. This can lead
to timeouts or use of a corrupt repository in some scenarios.
Those problems should no longer occur.
MozReview-Commit-ID: QAOSLyc0xD
--HG--
extra : rebase_source : c241626a5a59391b80baf34d50e90fc543e636f7
Since the buildbot-based Windows builds using releng.manifest are busted
anyways, there is no reason to keep clang entries in there. Which makes
those manifests identical to clang.manifest, so remote the latter.
--HG--
extra : rebase_source : eef7eca4bafc4e348eadc04d6da2bd17ea20deea
Currently manifestparser will only look for line continuations *after* looking for a key. This means
that line continuations cannot contain key separators. For example, this:
[test]
foo=
bar=baz
gets treated as:
{'name': 'test', 'foo': '', 'bar': 'baz'}
Here, bar=baz will be treated as a new key/value pair despite the indentation. This patch switches
the order around, so we look for a continuation first. Now, it is only treated as a continuation if
the indent is greater than the indent of the preceding key.
So this manifest:
[test]
foo=bar
baz=fleem
is a continuation and results in:
{'name': 'test', 'foo': 'bar\nbaz=fleem'}
But this manifest:
[test]
foo=bar
baz=fleem
is not a continuation, and yields:
{'name': 'test', 'foo': 'bar', 'baz': 'fleem'}
MozReview-Commit-ID: FAMP5TUIo9q
--HG--
extra : rebase_source : 624c53cfe0565374c1224dd86a3fffc8831279d3
Bug 1374748 removed Stylo-specific builds, but there are still a few config
files left behind that are now unused. This cleans them up.
MozReview-Commit-ID: EAUx7YKQBmN
--HG--
extra : rebase_source : 7e2124f7e5625d25efc5e868e151dbdc02cfba65
Ahem.ttf is copied to $(DIST)/bin/firefox/fonts, but this file is broken due to text mode copy. So we should use binary mode instead.
MozReview-Commit-ID: KP7yNyPiejU
--HG--
extra : rebase_source : 2de749f458a6d4650f9044f1912ff97835c5b795
mozharness used to call compare-locales with obj-l10n as MOZ_OBJDIR, while
calling configure and installers-% with obj-firefox.
Just make all the configs use the same objdir, instead of diving in
to mozharness.
MozReview-Commit-ID: 2CXTxVfYP82
--HG--
extra : rebase_source : 6a24927ef22701598f6807c4b7e0636172642655
The page load timeout of Marionette is 300s long. By using the same value
for the firefox ui harness invocation from mozharness it can cause a race
condition. Lets increase the value to what the Marionette jobs use to be
in sync.
MozReview-Commit-ID: FUevLmRIMSF
--HG--
extra : rebase_source : 43ee8b9009ba4ea89c8e8642d44eecf643b05ecc
Enable Stylo tests for win32, win64, and macosx64. Talos will be considered
separately.
MozReview-Commit-ID: BvTkddDX2Kk
--HG--
extra : rebase_source : 8d4df4469d4017b12bfbd33a98cc9f38111aa210
Flip around the logic of bug 1356122, so that the default from Stylo runs is the
parallel traversal, but we can opt in to single traversal as desired.
This ensures that for testing on other desktop platforms, we use parallel
traversal as the default.
MozReview-Commit-ID: KoBe1ltHP52
--HG--
extra : rebase_source : ad96f24c9760544c5812060f26e9ca18e5bf2ba8
This allows syntax like --try-test-paths web-platform-tests-reftests:path/to/test
MozReview-Commit-ID: uAet1ilPVy
--HG--
extra : rebase_source : 447277e47701435186ad87dfc089bd21f2bd1907
for: Land changes to flip windows nightlies on, and to tier 1 on central
MozReview-Commit-ID: H5Ysplr4NV7
--HG--
extra : rebase_source : 57b32b09d29ec3ced716dcd8e55bfa3bcb7c4475
Bug 1374940 adds a MOZ_TOOLCHAINS environment variable with a list of
path@task-id strings, where task-id is corresponding to the (possibly
optimized) toolchain job, and path corresponding to the
toolchain-artifact defined for that toolchain job.
We want to use that to pull artifacts instead of tooltool packages.
--HG--
extra : rebase_source : 277daa2c83d6d197975cb4ef36ee131176afa992
Land date changes to support windows nightlies onto central
MozReview-Commit-ID: DUg6pXQ8OiI
--HG--
extra : rebase_source : cf91a7ef950d706daaf4ccc8d5631cc24156c7ff
Land date changes to support windows nightlies onto central
MozReview-Commit-ID: 5un10JBaKrH
--HG--
extra : rebase_source : 45f61200a794f465637da657c0fa6e936a81bc09
Land date changes to support windows nightlies onto central
MozReview-Commit-ID: 4Nxhbutjhfu
--HG--
extra : rebase_source : 2a180738584d6de77fe1b8fe81ecb004582000c5
Land date changes to support windows nightlies onto central
This patch also allows us to not reconfigure to run repackage on windows, which saves us both in potential pain points and execution time.
MozReview-Commit-ID: xnz98Z5N06
--HG--
extra : rebase_source : 1630a3b6b646a83abeb05000e77e0e3c3238250b
Land date changes to support windows nightlies onto central
This will be needed for windows, because if the artifact is output to a folder that is, itself, an artifact folder, we'll end up publishing the named artifact twice. Which is unhelpful.
MozReview-Commit-ID: 5S6SNul6Fm9
--HG--
extra : rebase_source : 5501d8d757c7408c9dd014bc04a8d36429bdd9b5
Land date changes to support windows nightlies onto central
MozReview-Commit-ID: Jwk7c2W1QP7
--HG--
extra : rebase_source : c6f68a8b3621b3a743d1f1eb4a6857d23dd801b7
