Without this, "debug" and "opt" builds share the same cache. This
causes builds to randomly fail in automation.
MozReview-Commit-ID: Fend6GGmhFk
--HG--
extra : rebase_source : c3370938364b1e2cc94081d58ba9f35ce1060e5d
This fixes the use of the bogus pattern `taskcluster/**.py` which does not
actually match python files in subdirectories.
MozReview-Commit-ID: 3UqM5UQ0HOE
--HG--
extra : rebase_source : 73eec18034a11256273589b69b6d95aabceecd03
We've created a new worker type based on c4.4xlarge and m4.4xlarge
instances to replace the existing worker types based on c3.2xlarge,
m3.2xlarge, and r3.2xlarge. The new worker type has 16 vCPU instead
of 8, which makes build tasks complete ~10 minutes faster.
We've also consolidated down into a single pool for workers.
MozReview-Commit-ID: sOfKspFjCS
--HG--
extra : rebase_source : 8c7b030e0d9376dd9c07b71b8987be0c300fb691
This was recently refactored (incorrectly) in 147b245de122
(bug 1290282). Before that refactor, we only defined {{build_name}}
in base_linux64.yml. After, we hard-coded "linux32" and "linux64."
The change to linux64 resulted in ASAN builds sharing the same cache
name. Derp.
This commit restores {{build_name}} to the cache path. And for
consistency, we use {{build_name}} on both 32 and 64-bit variants.
MozReview-Commit-ID: KhqIjzARETO
--HG--
extra : rebase_source : 6b110302b66e052ec3d79bdc5b05a190cb0a7749
We run only a small subset of the test suite to save resources. Over
time, we'll run more of the platform test suite.
--HG--
extra : rebase_source : 264fbe55be5b56857278ae8d731ef51b1b41cf3c
extra : histedit_source : a3550c7e2ea8a41cd6285e511b7268a9e7af11cd%2Cec6024dadcb7c071170aa452064776666283bb27
MozReview-Commit-ID: Lrxi8t53nwy
If a developer adds '--rebuild N' to their try syntax they will get test jobs scheduled N times.
This is useful to determine intermittency rate.
This fixes a regression due to the recent refactoring on how we schedule tasks.
--HG--
extra : rebase_source : 355ca631353015bf63461c194168d753efd6958e
Without this, current umask may influence test results. That was
causing differences between automation and local runs.
MozReview-Commit-ID: 1eu613aBpKB
--HG--
extra : rebase_source : 41c92b9ea795217e715dfa949d3444534aafb7c7
Before, we were returning None, which gets converted to 0.
Derp.
Also fix a flake8 failure introduced by 9f5fbb3066c9.
We'll also need to generate a new decision image. But that will
require someone with TC privileges to be around. That can be done
in a separate commit to unblock this from landing and fixing
consumers of run-task that aren't the decision image.
MozReview-Commit-ID: 6XuoIxjDozF
--HG--
extra : amend_source : 4a3047d09d9bdce5a173a71ec3c68b8c4579a910
Before, we were returning None, which gets converted to 0.
Derp.
Also fix a flake8 failure introduced by 9f5fbb3066c9.
We'll also need to generate a new decision image. But that will
require someone with TC privileges to be around. That can be done
in a separate commit to unblock this from landing and fixing
consumers of run-task that aren't the decision image.
MozReview-Commit-ID: 6XuoIxjDozF
--HG--
extra : rebase_source : 836f4996cb9be24e19504f73792b159f0ffbdbee
Until now Gecko decision task artifacts would expire after a week. Unfortunately, some of these artifacts are
needed in order for Treeherder to determine which tasks it can schedule on a given push.
In this change, we're changing the expiration of all artifacts except those that are related to docker image
optimizations, which, are very large files and we only want to keep for a week.
MozReview-Commit-ID: 96ahyWmoI7W
--HG--
extra : rebase_source : d0f4d65e3b5bfae832a5aaed60c0dfd58f0bdc62
I was looking at some decision task logs and noticed lines like:
Connection pool is full, discarding connection: taskcluster
I also noticed lines like:
Starting new HTTP connection (153): taskcluster
In this case, we had established 153 TCP connections to a server.
Looking at the requests source code, a requests.session by default
creates a connection pool with capacity of 10. There are actually 2
components to the capacity: idle connections and active connections.
What appeared to be happening was we could obtain an idle connection,
use it, and then it would be discarded when put back in the idle
pool because the idle pool was at capacity. Furthermore, it also
appears that connections were sitting around waiting for a TCP
connection.
This commit uses a custom "adapter" with an increased pool size that
matches the concurrency level of the code issuing the HTTP requests.
This should increase the number of concurrent TCP connections / requests,
decreease the number of TCP connections being used overall, and make
decision tasks complete faster.
MozReview-Commit-ID: 6NDbz78TM2y
--HG--
extra : rebase_source : a48e1dae31a5e2af2bc554b544a9019d50dc784f
This commit does a lot. But it's really not too difficult to comprehend
once you focus on the final state, which is basically the same as the
"lint" image and derived tasks.
Before, the "decision" image contained a "checkout-gecko" script and
"run-action" and "run-decision" scripts. The latter 2 invoked the first
script.
The "run-task" script basically does what the combination of these
scripts were doing before. So we switch to it.
While we're here, we also replaced the custom Mercurial installation in
this image with the shared install-mercurial.sh script. The
system-setup.sh script for the decision image is now short and sweet.
The YAML files for tasks using this image have been updated to use
"run-task." We no longer have to pass an environment variable to hold
command arguments. So we revert to putting these arguments inline in
the task's command. Dustin never liked passing the arguments as
environment variables, so it should make him happy ;) We add
--log-no-times because "run-task" prefixes its own timestamps on output
lines.
The path to the Gecko checkout has been changed from
/home/worker/workspace to /home/worker/checkouts to match changes made
in bug 1289643.
Finally, since "checkout-gecko-and-run" is no longer used, we delete it.
The Docker image version has been bumped accordingly.
MozReview-Commit-ID: HAGuCz6668Q
--HG--
extra : rebase_source : 9586873c21d67f8ce869b7ceee221e8fa8775539
Before, we simply executed scripts inside Docker containers. This
frequently resulted in a wall of text with command output. It was
difficult to discern things like the time spent performing certain
actions.
Before, individual tasks had to drop permissions from the default
root user themselves. Dropping permissions isn't exactly a trivial
thing to do and a number of tasks didn't do it or did it wrong.
Before, we had a "checkout-gecko-and-run" script that kinda/sorta
did common activities for us. But it was written as a shell script
and doing advanced things was difficult.
This commit can be treated as a rewrite of "checkout-gecko-and-run"
as a Python script. But it also does a bit more. It prefixes output
with timestamps so we know how long operations took. It features more
robust argument parsing, so we can add new features more easily.
To prove the new wrapper script works, the lint image and all tasks
using it have been converted to use it.
MozReview-Commit-ID: 5d95u5Xebtq
--HG--
extra : rebase_source : 3a1d84782b01d7743e846bd0c04d7867813dd8a3
Before, it ignored mach logging settings passed via command line
arguments.
MozReview-Commit-ID: CNckQHrOdX9
--HG--
extra : rebase_source : b4b190e8e2d073e8932799c8d8bedf49b5f3cfd8
Use the source RIDEALONG_BUILDS value in the module under test so that changes
to that variable do not cause the test to fail.
MozReview-Commit-ID: EfHQ7baBziB
--HG--
extra : rebase_source : 1c52bf62709236db14a3ce318495891a2eb274f4
The existing hash was for an empty tarfile. Oops!
MozReview-Commit-ID: 1KOZxnDmoOH
--HG--
extra : rebase_source : 5d4db299dba80f98ba0383e88a1f4cfcb1dbcc70
Now that Docker image building is called from Python, we can start to
do advanced stuff with it.
With this commit, we switch from building Docker images directly from
the source directory ("the Docker way") to using our custom Docker image
build contexts.
The main advantage of this is that locally-built Docker images can now
use our custom Dockerfile syntax to include extra files in the build
context!
The code for building a Docker image from a context has been extracted
to its own standalone function. I have nefarious plans for this in the
future, such as the ability to override the FROM syntax to specify
URLs of images. This would allow us to host base images on our own
server, which removes a dependency on Docker Hub and improves
determinism, since images on Docker Hub change all the time.
MozReview-Commit-ID: 5lTdV8yEHkc
--HG--
extra : rebase_source : c374558b82d0d0302351ffbf3c82878c6663f40c
build.sh had been reduced to invoking `docker`. We move that invocation
to Python and remove build.sh. Long live build.sh!
MozReview-Commit-ID: FQBDJv4HSaU
--HG--
extra : rebase_source : 82979f2526ab1ab6dd5c25032b8d81074c797d94
We already had code for resolving the image registry and tag. We
refactored it slightly to be more useful then changed build.sh to
accept the tag as an argument.
At this point, build.sh is basically a wrapper around `docker`. But
there's a special case for executing custom "build.sh" files we
need to eliminate first...
MozReview-Commit-ID: A9HVvxgCdG2
--HG--
extra : rebase_source : 30a408860aea619813f32723fe960d1224b5dbc7
