Do not speak to github.com as part of CI. Do not open ourselves to a
MitM attack. Be more deterministic.
MozReview-Commit-ID: JRGbraQtqJi
--HG--
extra : rebase_source : b03c36f0c604995f396d49ab3d046637ef864613
It doesn't seem good to tie ourselves to the Gecko tooltool manifest for
building clang-cl; we want to stick with something we can update on
clang-cl's schedule, not Gecko's.
We cannot depend on a fixed location for cl.exe in a taskcluster world.
We therefore need to make our build-clang.py script accomodate relative
path names for cc/cxx and assume those are binaries that should be
looked up on PATH.
We also need to modify the Linux build script so that the virtualenv is
used to look up the new 'which' package.
There are a number of changes here:
- Taskcluster machines don't have MSVC installed, so we have to setup all
the MSVC infrastructure ourselves;
- Gecko contains useful Python packages to use, so we need to setup a
python virtualenv to have access to those. Fortunately, mach handles all
of that for us;
- We need to add build tools to our PATH, as they are not pre-installed;
- We need to define UPLOAD_PATH ourself.
The build-clang-windows.sh is basically an exact copy of the Linux one
at the moment; we'll fix it up for all the Windows-specific taskcluster
bits in a future commit.
332a08725ed0 changed the store path logic in a quick and crude manner.
The code could lead to multiple shared stores existing on a cache
if checkouts were in different parent directories.
This commit refactors the code to explicitly declare a path to the
shared hg store. This restores the behavior of ensuring there is only
a single shared store per cache.
MozReview-Commit-ID: 19Aa1QVrVQ8
--HG--
extra : rebase_source : f74c0ab9cba2bf18e335f24c86dd095335961723
This definitely isn't all of it. But I got the obvious chunks. Still
need to clean up mozharness.
MozReview-Commit-ID: JTZBydP3i2r
--HG--
extra : rebase_source : 782401359036751896ef6c153a81a06cb14031bb
We were seeing issues with the Mercurial working directory not being
pristine. While I can't reproduce this, I have a hunch it is due to
mixing and matching stores and checkouts in TaskCluster. For example,
if a worker supports running concurrent tasks and 2 tasks arrive at
the same time, the caches for the store and checkout may look like:
(store0, checkout0)
(store1, checkout1)
However, the next task may get:
(store1, checkout0)
This may confuse Mercurial.
This commit eliminates the "hg-shared" cache and places the shared
stores as a sibling directory of the checkout.
MozReview-Commit-ID: 8SzyS6wWf9C
--HG--
extra : rebase_source : 6205f3fe944a6ad2cb833a5a7c0ae5ba136eaea6
Currently, it is executed via another shell script (build.sh from
the desktop-build image). build.sh is going away and this will be
the main script called by run-task. It already has a shebang. So
make it executable.
MozReview-Commit-ID: GT4Ix4xdx88
--HG--
extra : rebase_source : e1298db624bedf67f00cec5237a5256624cccf62
Previously, SpiderMonkey tasks were using tc-vcs to clone the tooltool
repository so they could have access to tooltool.py. tooltool.py is
already in the execution environment courtesy of being part of the
source checkout. So use it from there.
One less dependency on a 3rd party service at run-time. One less
point for a MITM attack against our automation.
MozReview-Commit-ID: JmavaBG4vii
--HG--
extra : rebase_source : 5f6c56d1398268ceba867bb47645281465949246
When we switch to run-task in a subsequent commit, we can no longer
rely on cwd being /home/worker. So have build-linux.sh change the cwd
explicitly.
MozReview-Commit-ID: 5sYAjHCFi3s
--HG--
extra : rebase_source : 35e1ceafb6dc880f7f28ee152ae7c78d67ca2f29
Currently, build tasks produce a zip file artifact containing the
content of testing/mozharness. This zip file is downloaded and applied
by any test using mozharness (which is most of them).
Now that some tests have a source checkout, we don't need to download
a mozharness zip because we can just use the source checkout.
This commit teaches test-ubuntu.sh to accept a custom path to
mozharness files. If that path is defined, we skip downloading the
mozharness zip.
The taskgraph has been updated to pass the path of mozharness in the
checkout to test-ubuntu.sh when a source checkout is available.
MozReview-Commit-ID: 2P17Wx1oytI
--HG--
extra : rebase_source : a631a87737b253aece3bbc550ce58168212ae87c
extra : source : 21fdf73bbb17e34cfe00e372695c4f21e4ba3e6a
Same deal as the last commit except for config paths.
MozReview-Commit-ID: 3AXwc3i3Iqu
--HG--
extra : rebase_source : 588020b22853f669d9b9ec1b368d82f4b6f14a97
extra : source : dda84d1fb12b3416c81aab675c58b95ecb9f37cd
Mozharness scripts all live in "mozharness/scripts." So, we don't need
to redundantly define this path. Let's define the script as the relative
path within "mozharness/scripts" instead.
IMO this cosmetic improvement can stand on its own. However, the real
reason for this change is to pave the way for running mozharness from
a source checkout instead of downloading a zip archive of it.
MozReview-Commit-ID: FUnMiLlCcGK
--HG--
extra : rebase_source : 8e11f98dc2f63fa7d7fbd893d68e3c6d475a5759
extra : source : 11ffdc56f5becfeb2ee03423f67749f4baf1f3a7
The only difference in these files was the order that pulseaudio is
started and whether compiz is started. We rename test-ubuntu1604.sh
to test-ubuntu.sh, add some distro release detection, and add
some conditional branches so it works on both Ubuntu 12.04 and 16.04.
MozReview-Commit-ID: CaSfuDxss3d
--HG--
rename : taskcluster/scripts/tester/test-ubuntu1604.sh => taskcluster/scripts/tester/test-ubuntu.sh
extra : rebase_source : 29040c1cf7baedda0aaeff4bd788d5d400c127f1
extra : source : f227ea4d52fed84e3e682de0aa4cb8869539d645
Currently, build tasks produce a zip file artifact containing the
content of testing/mozharness. This zip file is downloaded and applied
by any test using mozharness (which is most of them).
Now that some tests have a source checkout, we don't need to download
a mozharness zip because we can just use the source checkout.
This commit teaches test-ubuntu.sh to accept a custom path to
mozharness files. If that path is defined, we skip downloading the
mozharness zip.
The taskgraph has been updated to pass the path of mozharness in the
checkout to test-ubuntu.sh when a source checkout is available.
MozReview-Commit-ID: 2P17Wx1oytI
--HG--
extra : rebase_source : 5d97079f054f3cd236a1c2a3039c3894207377f9
Same deal as the last commit except for config paths.
MozReview-Commit-ID: 3AXwc3i3Iqu
--HG--
extra : rebase_source : c249f77289e24aceaae73dbca0c994b949215814
Mozharness scripts all live in "mozharness/scripts." So, we don't need
to redundantly define this path. Let's define the script as the relative
path within "mozharness/scripts" instead.
IMO this cosmetic improvement can stand on its own. However, the real
reason for this change is to pave the way for running mozharness from
a source checkout instead of downloading a zip archive of it.
MozReview-Commit-ID: FUnMiLlCcGK
--HG--
extra : rebase_source : 3a0093f1995120264fed89dc4af94eea3554bfe5
The only difference in these files was the order that pulseaudio is
started and whether compiz is started. We rename test-ubuntu1604.sh
to test-ubuntu.sh, add some distro release detection, and add
some conditional branches so it works on both Ubuntu 12.04 and 16.04.
MozReview-Commit-ID: CaSfuDxss3d
--HG--
rename : taskcluster/scripts/tester/test-ubuntu1604.sh => taskcluster/scripts/tester/test-ubuntu.sh
extra : rebase_source : 2153d24fbf8208851a6df8728b8a820166278751
Robust checkout is the preferred method to clone a mercurial repository. This should
speed up the cloning process a bit and reduce storage size.
MozReview-Commit-ID: 92rcwMwRLYN
--HG--
extra : rebase_source : 33a2349d636a07ad1643798bf798bdb175c4f1ca
Sometimes xvfb will not start up with the current retry/delay settings. This will
attempt to retry more and delay for longer to ensure xvfb has started up. Common
pieces of this have been factored out into a recipe that all docker images can schare
that need this functionality.
MozReview-Commit-ID: BTXkJkBWLZX
--HG--
extra : rebase_source : 7554d7784768a65deb2c9ccb89586e5f77550e9f
Sometimes xvfb will not start up with the current retry/delay settings. This will
attempt to retry more and delay for longer to ensure xvfb has started up. Common
pieces of this have been factored out into a recipe that all docker images can schare
that need this functionality.
MozReview-Commit-ID: 2ww0eT3cIt6
--HG--
extra : rebase_source : f343f8a2bf8370657b63afdd9d739988bbb6261c
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
One thing run-task isn't doing is created the workspace. So this
functionality has been moved into test-ubuntu1204.sh and
test-ubuntu1604.sh.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : 264efc9b30e62927dece848b9b1eab7542cf2cba
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
extra : histedit_source : a5ae54a810718d446ded8bab510e6cd605562737
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
One thing run-task isn't doing is created the workspace. So this
functionality has been moved into test-ubuntu1204.sh and
test-ubuntu1604.sh.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : 75a937321d1850caebbb1eeaab42d04638ef75d9
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
There are some packages like 'requests' that are bundled in the mozharness venv, but not in the test
package. It would be easy to manually add these to sys.path in the mach bootstrap script, but it's
much nicer to simply activate this virtualenv in the first place.
MozReview-Commit-ID: 8xeJEIgUbLj
--HG--
extra : rebase_source : b87e5ef46041d9d5a89d419e97fe3a316de6c8c2
Normally we start Xvfb as a background task, then run the tests from the
same script. However, in interactive mode we were starting Xvfb, the script
would exit, and then we would potentially run tests later on from another
script. Unforunately this meant that Xvfb was dying with the first script
and tests would fail.
This patch runs Xvfb in a screen session so that it will still be available
later on when running an interactive shell.
MozReview-Commit-ID: EduVyglo2BG
--HG--
extra : rebase_source : 6e2c40ee16f80792f038fd581168e181a2c4bf51
This fixes a race condition between the 'test-linux.sh' process and the 'taskcluster-interactive-shell'
process in interactive tasks.
MozReview-Commit-ID: GhqKpq5pAtj
--HG--
extra : rebase_source : 67b756d0373432404a4f7cc928bac09fc3f82e8a
