This extracts the current logic for finding nodejs into its own module in mozbuild. Configure and ESLint then use it.
For ESLint, this will change the first location it looks for nodejs to be the .mozbuild directory.
Differential Revision: https://phabricator.services.mozilla.com/D6430
--HG--
extra : moz-landing-system : lando
Move all fields of nsISSLStatus to nsITransportSecurityProvider
Remove nsISSLStatus interface and definition
Update all code and test references to nsISSLStatus
Maintain ability to read in older version of serialized nsISSLStatus. This
is verified with psm_DeserializeCert gtest.
Differential Revision: https://phabricator.services.mozilla.com/D3704
--HG--
extra : moz-landing-system : lando
Bumping the Mercurial version to 4.7.1 to apply a fix for a
performance regression introduced in 4.7.
Differential Revision: https://phabricator.services.mozilla.com/D5193
--HG--
extra : rebase_source : b06eae5268fe7f38f85f8644da7f08516840a827
Automatic changes by ESLint, except for manual corrections for .xml files.
Differential Revision: https://phabricator.services.mozilla.com/D4439
--HG--
extra : moz-landing-system : lando
Prior to this patch, getHSTSPreloadList.js would queue an XHR for every preload
list candidate site. This meant that there would be ~50,000 requests in flight
simultaneously. Simply processing these requests caused them to all time out,
and no useful work was done. This patch resolves them in batches of 250 to
avoid this issue.
Differential Revision: https://phabricator.services.mozilla.com/D3622
--HG--
extra : source : e0f0d5824dd8aaaaf1395e569cec1806b028b12e
extra : amend_source : 62f1cd6fc49cbae39c7691c5712906f775862887
Prior to this patch, getHSTSPreloadList.js would queue an XHR for every preload
list candidate site. This meant that there would be ~50,000 requests in flight
simultaneously. Simply processing these requests caused them to all time out,
and no useful work was done. This patch resolves them in batches of 250 to
avoid this issue.
Differential Revision: https://phabricator.services.mozilla.com/D3622
--HG--
extra : moz-landing-system : lando
When the HSTS preload script was reworked to use async/await in bug 1436369,
`fetchstatus` would create an asynchronous xml http request and then attempt to
access a response header from it. However, there was nothing to ensure that the
request had completed before this code ran. This patch ensures that the request
has completed before the response header is used.
This patch also replaces a lingering instance of `Ci.nsISSLStatusProvider` that
should have been changed to `Ci.nsITransportSecurityInfo` in bug 1475647.
Finally, this patch removes the old, redundant getHSTSPreloadList.js in
security/manager/tools as well as the unused nsSTSPreloadList.errors file in
security/manager/ssl.
Differential Revision: https://phabricator.services.mozilla.com/D2807
--HG--
extra : moz-landing-system : lando
This commit points the `install_mercurial.sh` script at the
newest Mercurial distributions uploaded to tooltool.
Differential Revision: https://phabricator.services.mozilla.com/D2626
--HG--
extra : amend_source : a9cd500cc86106750aa5b5472e624a00128cac68
extra : histedit_source : 7a06d9c77b6f7a7b2110dc6bc15858c7aa3b7242
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
This will start to cause an error when a newer shellcheck is available in CI
Differential Revision: https://phabricator.services.mozilla.com/D1940
--HG--
extra : moz-landing-system : lando
1. Updated hgrepo to work with mozilla-beta, mozilla-esr60 and project branches (just in case)
2. Presquashed commits, so we only submit one.
3. Replaced 'which' with 'command -v' to avoid future shellcheck issues.
Differential Revision: https://phabricator.services.mozilla.com/D1582
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : source : 0b941cbdca76fb2fbb98dc5bbc1a0237c69954d0
extra : histedit_source : a3e43bdd8a9a58550bef02fec3be832ca304ea93
