We also include the example APK, since it will be helpful to be able
to regression test the example without downloading the AAR and
manually building the example with that AAR.
MozReview-Commit-ID: CMtA1FRS0Rf
--HG--
extra : rebase_source : 42bab43c69093bd008242ec96e74f53bde150583
We need to bump the Gradle Deps task, which fetches dependencies, to
include new test dependencies; and use freshly uploaded tooltool
archives (manually uploaded) containing the new test dependencies.
MozReview-Commit-ID: 8bNOVQPHlk6
--HG--
extra : rebase_source : 0c80117fb58e43f9c857027941f0a14f03b97f13
partial_env only works if the class inherits from some other base
class, which apparently not all callers of this method do. So just
pass a copy of the environ dict with PYTHONUNBUFFERED added.
MozReview-Commit-ID: Ag75x28NR4D
--HG--
extra : rebase_source : f19b2be2db0a4b321542cc353a4599481ba60146
We did the same thing for run-task in bug 1304964 because otherwise
output may get buffered, triggering "no output in N seconds"
failures in automation.
We're starting to execute mozharness scripts from source checkouts
in automation. This means we can stop downloading and extracting
files that are already available in the source checkout and just
reference files from the checkout.
Web platform tests (WPT) are a logical place to start because they
are pretty well isolated. This commit modifies the mozharness script
for WPT execution to use files from a source checkout (previous
commits have guaranteed that a pristine source checkout is available
to the test execution environment).
As part of this, we also need to define an explicit mozinfo.json
path because previously it was relying on parent directory traversal
to find it (WPT tests were under a directory containing the
mozinfo.json file).
MozReview-Commit-ID: C1dlKC4eSzr
--HG--
extra : rebase_source : af6f5ad0f88739efc5f5e0c74106a50e845564f6
4 WPT config files all contained the same config options boilerplate.
Inline it into the WPT mozharness script.
MozReview-Commit-ID: 5Lba8QeKMTA
--HG--
extra : rebase_source : a99f8e25d04a3e3344db1cf280fd79c47177ccb4
list.extend() is favored over +=. Also use single quotes, fixup
indentation, and factor out a common variable.
MozReview-Commit-ID: 3qVDGrkYhVe
--HG--
extra : rebase_source : 7573dca951eca0611c86cbb42e3cf53ccae078f4
We add a mozharness action to the TestingMixin base class that ensures
we're running from a VCS checkout and we add this action to the WPT
script.
This ensures that we always have a source checkout available, even
in buildbot. (Before, we only had a source checkout in TaskCluster.)
MozReview-Commit-ID: 26NxwDZywXr
--HG--
extra : rebase_source : 6aea0390b0c9ff43972ef6fe6592f104401cc3fc
Rather than clearing actions in volatile_config, add in actions from the artifact config's
default_actions. Incompatible actions are then skipped based on 'forced_artifact_build'
config value.
MozReview-Commit-ID: IZuDvxcQ7cN
--HG--
extra : rebase_source : 265f973959d031617beb11852bb51e7d5f90c8bc
Now that we've removed support for using easy_install, we no longer
need the "install_method" argument to specify how we want to install
packages since there is only one method: pip. So remove that code.
MozReview-Commit-ID: BmjerQtfHov
--HG--
extra : rebase_source : 44427108c5a043ed929747323ea539dcda10c1cb
Support for easy_install was added in bug 761809 as part of supporting
pywin32. We just removed support for pywin32. And there are no in-tree
consumers using the "easy_install" install method. Furthermore,
easy_install is effectively deprecated as a package install mechanism:
pip should always be used.
So, we remove support for easy_install from mozharness.
MozReview-Commit-ID: CN1meLukqY6
--HG--
extra : rebase_source : 883e427f0b5b634a519c3564dd31577e9b164414
pywin32 was removed as a requirement to run Talos in bug 726700,
~3 years ago. The references in mozharness were never updated,
apparently.
MozReview-Commit-ID: FMYxLCNa63H
--HG--
extra : rebase_source : 424b9b301a1c615acd3fd221df50e10a6c00d2cb
Proxxy is only configured in buildbot land. Don't enable it in
TaskCluster.
Ideally, we'd only enable proxxy if we detect we're in a buildbot
environment. But the change in this commit is more conservative
and aligns with existing behavior.
MozReview-Commit-ID: HBCdQ6MkYGa
--HG--
extra : rebase_source : 08c753d7af7a4ee95c557d9deb6401c4f2da4547
So we can detect when we're running on TaskCluster. This will
be used to adjust environment settings accordingly.
MozReview-Commit-ID: JEG1E3tWsc5
--HG--
extra : rebase_source : 2acb70bd9accbde44ccb8530002ba1e892b94ce2
Downloads geckodriver from tooltool when wdspec tests are being run,
and adds the --webdriver-binary argument
MozReview-Commit-ID: AJeP0YDk7Yl
--HG--
extra : rebase_source : 497f25c5af32b1851adf3a6f0b90a20640b6ccc6
Failing to find symbols in this case should be turned into a warning rather than dumping the traceback
since we're going to rely on mozcrash doing the right thing later on.
This will reduce unnecessary reporting of symbols not being available.
MozReview-Commit-ID: GXO01B7vzGV
--HG--
extra : rebase_source : 99ff82ffca6eed209ce6fd31ab747239d7100516
The virtualenv is placed in the "work dir" by default. If we
clobber the "work dir" at the beginning of the job, the parent
directory of the virtualenv may not exist and virtualenv creation
will fail because we set cwd to the work dir.
Fix that by ensuring the work dir / cwd always exists when
creating the virtualenv.
MozReview-Commit-ID: FAZPP1Sm22k
--HG--
extra : rebase_source : 126443cbcd5c83aeb47848bfc90ae28be9c9f596
From changeset 3282813aa892f0fc247181a33ce0bde2b751da50 from the
version-control-tools repo. File added without modifications.
Upstream change was peer reviewed.
Failing to find symbols in this case should be turned into a warning rather than dumping the traceback
since we're going to rely on mozcrash doing the right thing later on.
This will reduce unnecessary reporting of symbols not being available.
MozReview-Commit-ID: GXO01B7vzGV
--HG--
extra : rebase_source : 5fa15dcf89bedea2b4e6ff52f6d06461fe5e208d
download_unpack() is managing to download files correctly, however, sometimes we get an exception
that the zip file is corrupted.
This change adds more logging and saves the fetched file to disk in order to get uploaded as an artifact
for inspection.
MozReview-Commit-ID: 2KCK6qGNor4
--HG--
extra : rebase_source : 05f29616c90f36991582d285c6fa00d62fe06b40
In automation, we try to use pypi.pvt.build.mozilla.org nearly
everywhere. This hostname doesn't resolve in TaskCluster and
outside of buildbot automation.
A consequence of work in bug 1304176 and using a modern pip is
that we attempt to connect to all defined pip links. This was resulting
in pip attempting connections to pypi.pvt.build.mozilla.org. And
since pip was attempting retries, this resulted in a several
second delay for all pip operations if that host didn't resolve.
This commit adds a DNS lookup in mozharness before using a pip
--link. We spend a little bit of overhead in mozharness doing a
DNS lookup. In return, we guarantee we'll avoid a multiple second
pause if any links don't resolve. This is somewhat hacky.
But it seems like the easiest solution.
MozReview-Commit-ID: EecqBQSW75R
--HG--
extra : rebase_source : 9664a3694232e4ce2dec216b036ba29783c2842d
Check try message for --artifact even if fx_desktop_build.py is run with
--skip-buildbot-actions
We can't rely on buildbot config. Add checks to TryToolsMixin._extract_try_message so
that it works even if self.buildbot_config is None.
MozReview-Commit-ID: 1xErjuOArBe
--HG--
extra : rebase_source : 2f3204b37e67fd9a77dbff0fa93ab894b08181c1
From changeset 872711d144202b3f95e090a95f45cc1c45831caf from the
version-control-tools repo. File added without modifications.
Upstream change was peer reviewed.
In bug 1305752 we discovered that we download a zip file into memory without any issues, however,
when we tried to unzip we discovered that we have an invalid zip file.
The information in the logs is not sufficient to determine what could be the root issue.
MozReview-Commit-ID: DKwDUCmUhFF
--HG--
extra : rebase_source : 7b1ea32dcdc497831a387986d4449fffa37f54da
We're starting to execute mozharness scripts from source checkouts
in automation. This means we can stop downloading and extracting
files that are already available in the source checkout and just
reference files from the checkout.
Web platform tests (WPT) are a logical place to start because they
are pretty well isolated. This commit modifies the mozharness script
for WPT execution to use files from a source checkout (previous
commits have guaranteed that a pristine source checkout is available
to the test execution environment).
As part of this, we also need to define an explicit mozinfo.json
path because previously it was relying on parent directory traversal
to find it (WPT tests were under a directory containing the
mozinfo.json file).
MozReview-Commit-ID: C1dlKC4eSzr
--HG--
extra : rebase_source : fad7905abafaf126329aa25a96efafc40b01051f
4 WPT config files all contained the same config options boilerplate.
Inline it into the WPT mozharness script.
MozReview-Commit-ID: 5Lba8QeKMTA
--HG--
extra : rebase_source : c613db4107128ad2fd4ba568836ec27e0a9a92a8
list.extend() is favored over +=. Also use single quotes, fixup
indentation, and factor out a common variable.
MozReview-Commit-ID: 3qVDGrkYhVe
--HG--
extra : rebase_source : 0c5faa30f5be09165f61943300ecfb2e9c7080f4
We add a mozharness action to the TestingMixin base class that ensures
we're running from a VCS checkout and we add this action to the WPT
script.
This ensures that we always have a source checkout available, even
in buildbot. (Before, we only had a source checkout in TaskCluster.)
MozReview-Commit-ID: 26NxwDZywXr
--HG--
extra : rebase_source : d9c0fade450ab14c0b52be674c3c92bf670d2d3b
These are the only 2 definitions of the hg.mozilla.org certificate
fingerprint in mozilla-central. The certificate changed on
2016-09-26. So update the fingerprints.
This /might/ be a cargo cult because automation should be pinning
the hg.mozilla.org certificate everywhere. An alternative to this
commit would be to remove the fingerprint pinning from these
scripts. But if these scripts are run by humans, we may want to keep
the pinning in.
MozReview-Commit-ID: 4FUhAGc2oqx
--HG--
extra : rebase_source : fa8889ffbb70c14270acde67121192f7c1932258
If we don't catch HTTPError, the whole job fails since we get an uncaught exception.
MozReview-Commit-ID: 8jwW7ZSieyC
--HG--
extra : rebase_source : a184fe32bb73e786bd874a1c5f298d2b2c0bca83
We don't need a variable to hold the result. Just use return.
The "virtualenv_path" option has a default value, so it should always
be set. Add code confirming that. And refactor the code to use
less indentation. And remove a branch that can never occur since
the virtualenv path is guaranteed to be defined.
MozReview-Commit-ID: DZ6LnlxZJFj
--HG--
extra : rebase_source : 46682e9d33beb43e0b4fc181b9163afd373e7f70
Not sure why we support this. The code goes all the way back to the
import of mozharness 0.4 into the old mozharness Mercurial repo in
bug 651974.
Having fewer variations makes it easier to search for usage. So nuke
the variation.
MozReview-Commit-ID: IgwLMdvXGB0
--HG--
extra : rebase_source : 0f9e25b2d23a670154a024eee0264b545606cc80
The Python code is now intelligent enough to add this flag on the
command line if supported. Eliminate the copy pasta and help prevent
cargo culting.
MozReview-Commit-ID: H4rbjbbgtRd
--HG--
extra : rebase_source : 2f3acf666d73260ed97ad877f365b3091186f9e2
If mozharness is running from a source checkout, it has access to a
modern virtualenv+pip/setuptools vendored as part of the source
checkout.
This commit changes the virtualenv creation code to use the vendored
virtualenv when it is available.
A side effect of this change is that a modern version of pip will
now be used by mozharness when a source checkout is available. This
has a number of consequences.
First, modern versions of pip automatically create and cache wheels
when building packages. This should make automation faster since it can
now reuse cached wheels instead of having to download and rebuild
packages all the time.
Second, modern versions of pip support pinning package hashes. This
opens the door to use having more secure package downloads and more
determinism in our test environment.
Third, modern versions of pip require connections to package servers
be secure by default. Plaintext connections are disallowed by
default. A --trusted-host option or environment variable can be used
to override this behavior.
Since upgrading pip resulted in some jobs failing due to disallowed
connections to insecure servers, code to sniff the pip version and
add --trusted-host where it is needed/supported. This retains the
existing behavior. This is insecure. But fixing that is for another
bug.
As part of testing this, we were getting IOError inside virtualenv.py
when installing distutils:
IOError: [Errno 13] Permission denied: '/builds/slave/test/build/venv/lib/python2.7/distutils/__init__.py'
We worked around this by adding --always-copy to the virtualenv.py
invocation.
MozReview-Commit-ID: D29ao9ZASei
--HG--
extra : rebase_source : 031b2561a64ab1f89d25a3bfb8cf486a58b9f308
Now that we can detect when we're running from a source checkout,
we can start using things from source checkouts instead of relying
on host machine state or grabbing files from another server.
We start by using the vendored tooltool.py if available. This
avoids non-determinism. It avoids a possible 3rd party hosting
dependency on github.com. It avoids a possible MitM attack vector.
Wins all around.
MozReview-Commit-ID: L6hLveHZxBR
--HG--
extra : rebase_source : 67cc9d53fc0b3f92710ce41cc9f6556aa3ebbf99
We're going to start executing more mozharness scripts from a source
checkout. Rather than add config options to specify the location of
a source checkout - something that must be added to every mozharness
invocation - we teach BaseScript.__init__ to recognize when we're
running from a source checkout and set self.topsrcdir accordingly.
This will allow any script or class to check for self.topsrcdir
and change behavior accordingly.
MozReview-Commit-ID: 3uxOjol7ntR
--HG--
extra : rebase_source : 40795fe231a908b42a13581db3ee079c13138412
Without this, process output is buffered by default. This means
timestamps that mozharness prefixes to process output aren't
accurate unless the process is spewing enough output to flush the
output buffer.
Output buffering could lead to bad things. For example, a process
could emit output that would cause mozharness's output monitor to
abort the process. However, if that output is caught in limbo in
the output buffer, mozharness may take several seconds or even
minutes to react.
With this change, the mozharness process receives process output
as soon as that process writes to its standard file descriptors.
Once a newline is seen, mozharness will process it immediately.
Note that this only impacts the case where there is no output
timeout, as the existing code for output timeout uses mozprocess
and I'm pretty sure mozharness doesn't buffer output.
MozReview-Commit-ID: HBkYnfEw7Hb
--HG--
extra : rebase_source : e17b44d88f27c16b054a64c3cc2b3415297daf3b