Whenever an exception is raised while tests are executed, the log error message should only be
printed once. As best this should happen in `cli()`, so that subclasses can better set their
own behavior, and we safe us from re-raising the exception.
MozReview-Commit-ID: 5NLBnJAjUMQ
--HG--
extra : rebase_source : 17e1574c8671037912d85c0575db493c96f972b2
The robustcheckout Mercurial extension does a clone+checkout optimally.
Read the bug for more on it.
robustcheckout is already used by mozharness automation. It has resulted
in a significant reduction in I/O usage and utilization in automation.
This commit replaces tc-vcs with the robustcheckout equivalent.
We replace the existing tc-vcs scope and cache with a new one.
Because Dustin and I are paranoid, we maintain separate caches per
SCM level - even though we could arguably share the same cache. Defense
in depth.
Robustcheckout (when used with --sharebase) pools storage for related
repos automatically. i.e. changesets from inbound and central will
be in the same store. This means you likely only have one copy of
each changeset per cache. This can result in significant space savings.
And, since there are fewer copies floating around, hg.mozilla.org
and various network appliances are working less too!
Since tc-vcs is no longer used, we stop it from being installed.
While we're here, we also change the images to execute as the
"worker" user. This happens automatically as a result of using
the "checkout-and-run" script.
MozReview-Commit-ID: EDeebuP7TkT
--HG--
extra : rebase_source : 2bec5dd9d6fe5565831bb35f195859aa12dd0bf2
extra : intermediate-source : 06481d97a485f6566554b087bc3880d76361e8ec
extra : source : d368700c93ef085325a081219d7aeb8512bc54a1
extra : histedit_source : c07505273fc8f10acf8e8d3ee01e327afd0aa63d
The script will be used as the main command in task YAML files.
It changes ownership of caches. Then switches to the "worker" user.
Then performs a Gecko checkout. Then executes whatever command was
requested via its arguments.
The script has been added to the shared recipes directory so it can
eventually be used by other Docker images. This means if we e.g. want
to add Git support, we only need to update one file in the tree.
MozReview-Commit-ID: Fuy1VrdSGYn
--HG--
extra : rebase_source : 407b2c584d56c95e9d9b23781539f2979a775893
extra : histedit_source : bd8b7fd541ed27da31082730ad3054b68b06544b
Like we do for the decision image, we install Mercurial 3.8.4 from deb
files hosted on tooltool. This provides more control and determinism
than installing via apt.
As part of this change, Mercurial is upgraded from whatever was hosted
in apt to 3.8.4.
Since the deb packages don't provide a global hgrc, we create one
ourselves. This is effectively copied from the decision image.
Most of the work is being done in a new, standalone
install-mercurial.sh script. This script is part of the
newly-established testing/docker/recipes directory. The intent of this
directory is to hold common files referenced by multiple images. Our
custom Dockerfile syntax to include files from outside the directory
with the Dockerfile is used to add these files to the build context.
MozReview-Commit-ID: K7gVm2Geihj
--HG--
extra : rebase_source : 6d1089ac34e43d399c7cf608d09eaaf405df00f7
extra : histedit_source : 656a4cea33ef913102b03238475461884c2749a0
Using our special Dockerfile syntax to include arbitrary files, we
include the previously vendored tooltool.py file in the image build
context and add it directly from there. No github.com communication
needed.
MozReview-Commit-ID: J42iXj87LEu
--HG--
extra : rebase_source : 90845e6793629b56998bf2fae2985913ee49c4eb
extra : histedit_source : 1fd5e64e40ae700efcf78b54e2a865b0594e0955
Visual aligning makes diffs harder to read. Use line continuations
to avoid this. Also make the package list alphabetical.
MozReview-Commit-ID: KqT4aqYyZfH
--HG--
extra : rebase_source : 08d2e4f61860bf6183ec3afaf598be158cd182be
extra : histedit_source : ff450a22617425214e90d42a6f1b530da8682847
Changes to the decision Docker image have been compelted. We're ready to
use the new image.
We tag the image, update version references, change the task caches
so the new Mercurial pooled storage from the robustcheckout extension is
used, and convert the decision tasks to run as the "worker" user.
MozReview-Commit-ID: 61v9Ivy59zG
--HG--
extra : rebase_source : 640318a87660950c5e0680867a1bfdd68e35f127
extra : histedit_source : ec53fc576c00e5f2053167b37544ac7afccaecb5
When we switch to use robustcheckout for version control foo, we'll
also be taking the opportunity to have the decision and action tasks
execute as the "worker" user.
Since caches are mounted and owned by root and since tasks initially
run as root, this makes defining the container command in YAML a bit
difficult because we have to do some work as root then switch users
and continue executing. Rather than shoehorning all that complicated
logic into YAML, we introduce bash scripts that do it. These will
be plugged into the task YAML when we formally switch the tasks
to use the new Docker image.
We provide one script for running Gecko decision tasks. We provide
another for running action tasks. These are the two consumers of
the decision image we care about.
We also sneak in a change to add the executable bit to checkout-gecko.
MozReview-Commit-ID: CXlyHZJSHcP
--HG--
extra : rebase_source : 80621d4833a9d745eaff7da4641dfd4ace8ae1db
extra : histedit_source : e6ce7de5d14c8781d8dd94a8eff76c3227cd18b5
Now that Mercurial 3.8.4 and robustcheckout are in place, we convert
checkout-gecko from tc-vcs to robustcheckout.
As part of this, we remove references to tc-vcs from the Docker image.
This completes our changes to the decision Docker image. Image size has
been reduced from ~725 MB to ~217 MB. Not bad.
MozReview-Commit-ID: Hx9d02Al1TP
--HG--
extra : rebase_source : 05114e4e0e7fbbab2c89f25074abfeb7b9ba62ef
extra : histedit_source : 193c0bbb64cc1e468b5d7bb969d7f74e25947bde
web.cacerts matches what the Ubuntu package does by default.
[progress] changes are to make output in TaskCluster logs less
spammy (only 1 update per second instead of up to 10).
The robustcheckout extension will be used in a subsequent commit to
handle repository checkouts.
MozReview-Commit-ID: 2PvW4wEGk2u
--HG--
extra : rebase_source : 742627ba823d4f2097a4273e6cc6af8bb842c69f
extra : histedit_source : d479c1923c71605e9511e877b4b90d3b4d42f542
Previously, we were downloading tooltool.py from github.com. There
were a few problems with this.
First, there is a dependency on a 3rd party service. While the Docker
image should be cached, as a matter of principle we don't like hitting
3rd party services in our automation. The file is small enough, so we
just vendor it.
Second - and more importantly - we weren't validating the integrity of
the downloaded file. This means that a MiTM could possibly alter the
content of the file without us knowing (they would need a valid CA but
since the Ubuntu trusted CA bundle contains a lot of CAs from e.g.
governments, this isn't out of the question). Vendoring the file removes
this risk.
Third, behavior wasn't deterministic over time. We were always
downloading the "master" revision of the file. I like determinism over
time. Vendoring makes things deterministic.
MozReview-Commit-ID: 4DdSd42BnAu
--HG--
extra : rebase_source : cf73d2741fc186bebf06233efefdf85cd8cea3f2
extra : histedit_source : 76c7d81266a72010a9969ea32ac13c7bce2a0601
I'm not sure why the decision image has so many packages installed.
Most of them don't need to exist because the decision image only
needs to obtain a copy of the Firefox repo and run `mach`. This
doesn't require any build system per se. And all the Python
dependencies are vendored in the Firefox repo. All we need is a
Python 2.7 interpreter.
This change reduces the decision image size from ~700 MB to ~300 MB.
MozReview-Commit-ID: CUqc5TUVZSc
--HG--
extra : rebase_source : 5a2b3888b4c54c29bc8c8b9215ce36a4340574e5
extra : histedit_source : 61e70b06b703c3262ae1bc2f527f1919a3f450ec
We change the installation of Mercurial from via peep to .deb files in
tooltool. The .deb files were produced by Mercurial's built-in make
targets to produce .deb packages.
As part of this, we upgrade to Mercurial 3.8.4. It should be a drop-in
replacement.
Since we no longer use peep, we stop installing it and pip/setuptools
since they were only needed to run peep.
It's worth noting that we choose to install from .deb files instead of
pip because this keeps image creation small and simple. Otherwise we'd
have to install a compiler, etc.
MozReview-Commit-ID: INnKDHkX2uk
--HG--
extra : rebase_source : 0c6f30ff193dba5fbb5d90603e00f8be02816f9d
extra : histedit_source : 2afd18a694447bd133c26b7ccd562cdf7453b674
We're currently running Ubuntu 14.04 in the decision image. While it is
still in LTS support, 16.04 ships with a modern, properly configured Python
2.7. So we upgrade to 16.04 and drop the install of Python from source
because it is no longer needed.
This is part 1 of a larger refactor to this image.
MozReview-Commit-ID: CTbsPmTjcgs
--HG--
extra : rebase_source : eca12e98c8ff63cb302ea580da9296bd4cf31a4f
extra : histedit_source : 1a40405a9360239bf95d368c43ccfd0681609500
In preparation for running tasks as the worker user.
MozReview-Commit-ID: DLgD0lh5V2C
--HG--
extra : rebase_source : 1508517f9fbc986ada96cbe4ee77847ad6e1afcc
extra : histedit_source : 4b2957c47fcab8704416748613e7ff5badc61897
note: this requires a clean work dir unfortunately. so you have to blow away
the fake build/hg-share and any repos in build/
MozReview-Commit-ID: 3TfNLdga9Dt
--HG--
extra : rebase_source : 25972c5b53eb1bddd490c7aea6a085b713ff7d03
extra : amend_source : 5841fb61e94ab9c4c0f43b344f1a68d589a5c356
this can be uplifted through 48 mozilla-release
MozReview-Commit-ID: KncTJ8hAgfO
--HG--
extra : rebase_source : 18dc6c107a138317f95b433e33bf3081166c4478
Instead of checking for the paint time of the tspaint_test.html content,
we're now measuring the delta between process start and first paint as
reported by the parent process's startup info.
MozReview-Commit-ID: 868mf2vazwL
--HG--
extra : rebase_source : 7ba8062be91ca00ab1bbcd386b4c9213148e41f7
HSTS priming changes the order of mixed-content blocking and HSTS
upgrades, and adds a priming request to check if a mixed-content load is
accesible over HTTPS and the server supports upgrading via the
Strict-Transport-Security header.
Every call site that uses AsyncOpen2 passes through the mixed-content
blocker, and has a LoadInfo. If the mixed-content blocker marks the load as
needing HSTS priming, nsHttpChannel will build and send an HSTS priming
request on the same URI with the scheme upgraded to HTTPS. If the server
allows the upgrade, then channel performs an internal redirect to the HTTPS URI,
otherwise use the result of mixed-content blocker to allow or block the
load.
nsISiteSecurityService adds an optional boolean out parameter to
determine if the HSTS state is already cached for negative assertions.
If the host has been probed within the previous 24 hours, no HSTS
priming check will be sent.
(r=ckerschb,r=mayhemer,r=jld,r=smaug,r=dkeeler,r=jmaher,p=ally)
With the initial browser defaulting to remote, there's a greater likelihood
that the DOMContentLoaded event that is handled in the "get" function will
be fired by the initial about:blank instead of the actual desired page.
get() currently works around this by ensuring that the URL of the loaded
page matches the requested one when DOMContentLoaded fires. Unfortunately,
this doesn't work for pages that redirect via their HTTP headers (and will
therefore not fire DOMContentLoaded).
This patch fixes things by adding an nsIWebProgressListener that ensures
that the requested page has started to load before paying any attention
to the DOMContentLoaded events. This handles the redirect case. We also
compare against nsIChannel.originalURI for the about: redirect case.
For neterror pages (which never open channels, and therefore are not
seen by the nsIWebProgressListener), we just check that the page that
we attempted to reach was the one that was requested.
MozReview-Commit-ID: Gbbmfwat46s
--HG--
extra : rebase_source : 1848cd67757be8780f9e50253dc0ee1131467257
Before, it was assumed that the next load was the one that the Marionette client had
asked for, when this might not be the case. For example, when a new window opens,
it's possible for the initial about:blank load to be fired in content after the
parent has asked for a page to be loaded.
MozReview-Commit-ID: GPoJgbCvSju
--HG--
extra : rebase_source : 7b4c1638c2fe81a0a37d061a655e35aed0e2daa0
extra : source : b2e910bb1d726562548eba1148a81ec37300fb7b
