If mozharness is running from a source checkout, it has access to a
modern virtualenv+pip/setuptools vendored as part of the source
checkout.
This commit changes the virtualenv creation code to use the vendored
virtualenv when it is available.
A side effect of this change is that a modern version of pip will
now be used by mozharness when a source checkout is available. This
has a number of consequences.
First, modern versions of pip automatically create and cache wheels
when building packages. This should make automation faster since it can
now reuse cached wheels instead of having to download and rebuild
packages all the time.
Second, modern versions of pip support pinning package hashes. This
opens the door to use having more secure package downloads and more
determinism in our test environment.
Third, modern versions of pip require connections to package servers
be secure by default. Plaintext connections are disallowed by
default. A --trusted-host option or environment variable can be used
to override this behavior.
Since upgrading pip resulted in some jobs failing due to disallowed
connections to insecure servers, code to sniff the pip version and
add --trusted-host where it is needed/supported. This retains the
existing behavior. This is insecure. But fixing that is for another
bug.
As part of testing this, we were getting IOError inside virtualenv.py
when installing distutils:
IOError: [Errno 13] Permission denied: '/builds/slave/test/build/venv/lib/python2.7/distutils/__init__.py'
We worked around this by adding --always-copy to the virtualenv.py
invocation.
MozReview-Commit-ID: D29ao9ZASei
--HG--
extra : rebase_source : 031b2561a64ab1f89d25a3bfb8cf486a58b9f308
Now that we can detect when we're running from a source checkout,
we can start using things from source checkouts instead of relying
on host machine state or grabbing files from another server.
We start by using the vendored tooltool.py if available. This
avoids non-determinism. It avoids a possible 3rd party hosting
dependency on github.com. It avoids a possible MitM attack vector.
Wins all around.
MozReview-Commit-ID: L6hLveHZxBR
--HG--
extra : rebase_source : 67cc9d53fc0b3f92710ce41cc9f6556aa3ebbf99
We're going to start executing more mozharness scripts from a source
checkout. Rather than add config options to specify the location of
a source checkout - something that must be added to every mozharness
invocation - we teach BaseScript.__init__ to recognize when we're
running from a source checkout and set self.topsrcdir accordingly.
This will allow any script or class to check for self.topsrcdir
and change behavior accordingly.
MozReview-Commit-ID: 3uxOjol7ntR
--HG--
extra : rebase_source : 40795fe231a908b42a13581db3ee079c13138412
Without this, process output is buffered by default. This means
timestamps that mozharness prefixes to process output aren't
accurate unless the process is spewing enough output to flush the
output buffer.
Output buffering could lead to bad things. For example, a process
could emit output that would cause mozharness's output monitor to
abort the process. However, if that output is caught in limbo in
the output buffer, mozharness may take several seconds or even
minutes to react.
With this change, the mozharness process receives process output
as soon as that process writes to its standard file descriptors.
Once a newline is seen, mozharness will process it immediately.
Note that this only impacts the case where there is no output
timeout, as the existing code for output timeout uses mozprocess
and I'm pretty sure mozharness doesn't buffer output.
MozReview-Commit-ID: HBkYnfEw7Hb
--HG--
extra : rebase_source : e17b44d88f27c16b054a64c3cc2b3415297daf3b
These builds can be run on taskcluster to obtain per-test (JSDebugger) code coverage with the linux64-jsdcov build and overall (GCOV) code coverage with the linux64-ccov build. The linux64-jsdcov build also needed to have leak checking disabled for debug mode.
MozReview-Commit-ID: ASgrU2X7RQV
--HG--
extra : rebase_source : b2098e4d01039edd6cff37f3e6a26c2ed3d3d3ba
These builds can be run on taskcluster to obtain per-test (JSDebugger) code coverage with the linux64-jsdcov build and overall (GCOV) code coverage with the linux64-ccov build. The linux64-jsdcov build also needed to have leak checking disabled for debug mode.
MozReview-Commit-ID: ASgrU2X7RQV
--HG--
extra : rebase_source : af40a6e582665ffcb575092586731f595a362ae4
We believe we're getting incomplete bytes when fetching files from S3 to our EC2 instances.
This patch should help give us more information and retry few times before failing.
MozReview-Commit-ID: 7tUzZmS8Zph
--HG--
extra : rebase_source : f8c052c92d3ccccf18daf6cbf9832d8ec48a6ecd
Override actions specified in TC task definition with default_actions from
artifact config
Clear MOZ_SIGN_CMD when enable_signed is set to False.
MozReview-Commit-ID: FmI6lu5EKWg
--HG--
extra : rebase_source : 61489dc7de6652d7739c4a5179c1431334ae78c0
Override actions specified in TC task definition with default_actions from
artifact config
Clear MOZ_SIGN_CMD when enable_signed is set to False.
MozReview-Commit-ID: FmI6lu5EKWg
--HG--
extra : rebase_source : 0b21544b53e8fc3323ec0063f83cccb4a55f0014
Modify the mozharness script for the Marionette harness tests
to use the new pytest_mozlog plugin, so that the logs produced
will be parseable by Treeherder.
Change the arguments passed to `pytest.main` to instruct pytest_mozlog
to write log files in TBPL, mach, and raw (structured log) formats.
Read the TBPL log into the mozharness script log itself,
and upload all pytest-mozlog log files to the default
blob upload dir as specified in the Mozharness BaseScript.
MozReview-Commit-ID: HoEK2M1XGwN
--HG--
extra : rebase_source : 9042d89c6ab2befefe945fdcfd6999c35b9d7340
Currently 'stop-emulator' is its own action in mozharness. This means that anytime the mozharness
script finishes emulators will get shutdown. Usually, this is what we want to do. But if the user passes
in --no-run-tests, then presumably they are intending to run tests at a later time for some reason.
But when they do this, the emulator will be terminated, and then running tests later no longer works.
The main place where this becomes a problem is on interactive loaners in taskcluster. This patch fixes
the problem by instead running 'stop-emulator' only after the 'run-tests' step finishes. So if
--no-run-tests is passed in, the emulator will be left open.
MozReview-Commit-ID: 4lsKGpizfH7
--HG--
extra : rebase_source : e29733a4a69d4d063f225754d2ad8886e22b4782
Instead of downloading a file first and then unpacking it, we would like to
fetch it into memory and then unpacking directly from there.
This saves writing the file first to disk, thus, saving on IO.
MozReview-Commit-ID: JdNGnxIYEvy
--HG--
extra : rebase_source : 73c280fd23f3066d04dd3af07ac443db77dfd6d0
This new version of the schema validates that numeric values are
within an acceptable range (i.e. are not infinite)
MozReview-Commit-ID: 32oT39Bfcwg
--HG--
extra : rebase_source : 78ef5402e47e05f3f4e5a1db7b36e2d365c7b13b
