Fetches no longer need to be artifacts exposed via a 'fetch' task, they can
also be artifacts from a task's dependencies. The new format is:
fetches:
fetch:
- fetch-artifact-1.zip
- fetch-artifact-2.zip
build:
- build-artifact-1.zip
...
Specifying 'build' artifacts to fetch will error out if the task doesn't have
any build dependencies.
The 'fetch' key works the same as before, but it is now a special case. Unlike
'build' (or other dependencies), adding a fetch task's artifact here will
implicitly make our task depend on the corresponding fetch task. It will not
be an error.
Depends on D2028.
Differential Revision: https://phabricator.services.mozilla.com/D2102
--HG--
extra : moz-landing-system : lando
Fetches no longer need to be artifacts exposed via a 'fetch' task, they can
also be artifacts from a task's dependencies. The new format is:
fetches:
fetch:
- fetch-artifact-1.zip
- fetch-artifact-2.zip
build:
- build-artifact-1.zip
...
Specifying 'build' artifacts to fetch will error out if the task doesn't have
any build dependencies.
The 'fetch' key works the same as before, but it is now a special case. Unlike
'build' (or other dependencies), adding a fetch task's artifact here will
implicitly make our task depend on the corresponding fetch task. It will not
be an error.
Depends on D2028.
Differential Revision: https://phabricator.services.mozilla.com/D2102
--HG--
extra : moz-landing-system : lando
The 'use_fetches' transform is currently only being used by toolchain tasks,
but we'd like to expand this to more kinds (like 'test' and 'source_test').
The problem is that 'use_fetches' doesn't have a schema, and assumes things
about the kinds of keys that will be set in the job. For example, it assumes
that job['worker']['env'] is going to be forwarded up to the jobdesc properly.
By moving this transform into the set applied to all 'job' tasks, we:
A) Have a task schema we can reliably depend on
B) Can automatically use it from any 'job' task without kind specific
modifications
Since the toolchain tasks apply the 'job' transforms (almost) right after
the 'use_fetches' transform, this change just works.
Differential Revision: https://phabricator.services.mozilla.com/D2028
--HG--
extra : moz-landing-system : lando
This patch makes the QR test platforms tier-1 by default, and removes
the ad-hoc bits that were making individual QR jobs tier-1 before.
However, it also explicitly downgrades some QR jobs to tier-2 or tier-3;
comments in the yml files indicate why.
MozReview-Commit-ID: 1UfPuhcMvIW
--HG--
extra : rebase_source : a2347f6a5929246aaba7656b59c0b8f7aa4ca081
This is needed to not have a circular kind dependency when we actually spell out all dependencies (in a following patch)
Differential Revision: https://phabricator.services.mozilla.com/D1695
--HG--
rename : taskcluster/ci/repackage-signing/kind.yml => taskcluster/ci/repackage-signing-l10n/kind.yml
extra : rebase_source : c2998ba23f213090d27495eb44c3bde3a1628dff
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : a17aa496bf3d4af4d1349d69a637c686c6817d0f
Also include webgl2-deqp, which we would like to run eventually, but not yet.
MozReview-Commit-ID: CY4hYCI95ws
--HG--
extra : rebase_source : 9973df0f905bb65d2e8b8c66a6a57e8869e527c1
Also include webgl2-deqp, which we would like to run eventually, but not yet.
MozReview-Commit-ID: FDWdu1J0end
--HG--
extra : rebase_source : a47d88cb2c5eb82e4dfaa9e58d76acbf0736d35d
We shouldn't run this on central, as it falls back to the dev configs, and fails.
It should be fine on beta/release/esr60. I had to move this version of the check to its own
kind to avoid the dependency tree bringing in the entire build process. Perhaps we can
refactor later to avoid duplication
Differential Revision: https://phabricator.services.mozilla.com/D1765
--HG--
rename : taskcluster/ci/release-bouncer-check/kind.yml => taskcluster/ci/bouncer-check/kind.yml
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : c788ef4f7da9949b81df2f0577af6f6039ea63d8
Having to define them explicitly feels too redundant for my liking.
I believe we didn't do this before because we were defining things
in terms of "using: fetch-url." Now that we are using a custom
transform, we can have nice things.
Differential Revision: https://phabricator.services.mozilla.com/D1575
--HG--
extra : rebase_source : e4b08bf799f4011904df0194035527ec625123dd
extra : intermediate-source : 3bb31648056bf9d7760c59420d26de25fbf2b268
extra : source : 52c528fe192df9b1704f4d24274b2adb435763ce
Previously, the fetch kind was defined by a job "using" flavor.
An upcoming commit will need to derive new tasks from fetch job
definitions. In order to do this, we require a transform. And
this transform would need access to the original data from the
job description. But by the time the "using" transform runs, a
lot of this data is thrown away. It would be possible to stuff
the meaningful metadata inside attributes on the created task.
But this would result in the fetch logic being fragmented across
multiple Python modules (a fetch-specific transform and a job/using-
specific transform). I think it is better to keep all that logic
in a single Python module.
This commit converts the "using: fetch-url" job transform to a
dedicated transform for the fetch kind.
Since we're now using a dedicated transform, we no longer need
to use the normal job schema for defining fetch jobs. So we
refactor the schema a little so it is simpler.
I verified the taskgraph output is nearly identical to before by
diffing the JSON output of `mach taskgraph full`. Aside from the
additions of ['worker']['implementation'] and ['worker']['os']
fields (which seem to be required by a schema somewhere), everything
was identical.
Differential Revision: https://phabricator.services.mozilla.com/D1574
--HG--
rename : taskcluster/taskgraph/transforms/job/fetch.py => taskcluster/taskgraph/transforms/fetch.py
extra : source : d22981772dd995a492b5fdb67c887d55d24654db
extra : amend_source : ab472fd2a8d652092814443b8da94a580a5bfd53
Having to define them explicitly feels too redundant for my liking.
I believe we didn't do this before because we were defining things
in terms of "using: fetch-url." Now that we are using a custom
transform, we can have nice things.
Differential Revision: https://phabricator.services.mozilla.com/D1575
--HG--
extra : rebase_source : 74639800c735d2c90b110b5c378eed570fdc3863
extra : amend_source : 92364ad0061c87bf111f6f11ba5bf39d8bfbe619
extra : intermediate-source : 825f0caef542f1b33376fbcc81dc18b915bf218b
extra : source : 52c528fe192df9b1704f4d24274b2adb435763ce
Previously, the fetch kind was defined by a job "using" flavor.
An upcoming commit will need to derive new tasks from fetch job
definitions. In order to do this, we require a transform. And
this transform would need access to the original data from the
job description. But by the time the "using" transform runs, a
lot of this data is thrown away. It would be possible to stuff
the meaningful metadata inside attributes on the created task.
But this would result in the fetch logic being fragmented across
multiple Python modules (a fetch-specific transform and a job/using-
specific transform). I think it is better to keep all that logic
in a single Python module.
This commit converts the "using: fetch-url" job transform to a
dedicated transform for the fetch kind.
Since we're now using a dedicated transform, we no longer need
to use the normal job schema for defining fetch jobs. So we
refactor the schema a little so it is simpler.
I verified the taskgraph output is nearly identical to before by
diffing the JSON output of `mach taskgraph full`. Aside from the
additions of ['worker']['implementation'] and ['worker']['os']
fields (which seem to be required by a schema somewhere), everything
was identical.
Differential Revision: https://phabricator.services.mozilla.com/D1574
--HG--
rename : taskcluster/taskgraph/transforms/job/fetch.py => taskcluster/taskgraph/transforms/fetch.py
extra : rebase_source : 7186d4b8fe2c6a8dc7dfa7629dfb96ec670411e9
extra : amend_source : bce1f37233d7496841a8a7aa0eaf89542bbba610
The max-run-time for tasks appears to have been mostly cargo
culted. In particular, a value of 36000 (10 hours) is quite absurd:
no single task takes that long to execute.
This commit reduces the max-run-time of several tasks to
more reasonable values. The goal here is to prevent
excessively long-running tasks. There is definitely room to
further tweak values to further mitigate long-running tasks. But
let's bite off the biggest chunk first, since that doesn't
require much mental effort.
There is a possibility I overshot on some of these tasks. If we
get timeouts, we can always increase the timeout again.
Differential Revision: https://phabricator.services.mozilla.com/D1716
The by-project and by-build-platform are collapsed by resolve_keyed_by(), and in turn by keymatch() in
utils/attributes.py. That needs a regexp to look for matches (if a simple match isn't possible). Rev
39eed777c0c4 missed that when it landed. The trailing hypens are to distringuish between the two linux
platforms.
MozReview-Commit-ID: LiAJpfV71Ws
--HG--
extra : rebase_source : 6952e4b6be4b05cc7917efba8a346211d1e5c7e0
Instead of clang 4, which they were the last to use, so remove the
clang 4 toolchain.
--HG--
extra : rebase_source : d03a083e9217aeb6c1d2c91decb978426f0e8d1a
For those toolchains built with clang, use clang 6.
The only jobs remaining to use the clang 3.9 toolchain are the
base-toolchains build jobs.
--HG--
extra : rebase_source : 1fe92e9e2730c11ce5ce87dddd6496228c3d270c
Many builds have been using clang 6 explicitly because they needed
features or fixes from it that weren't available in earlier versions.
Now that other builds have kept up, it's probably desirable to keep
everything in sync.
--HG--
extra : rebase_source : deb9daaf792d05518e17b1c48589a3b33035a7ab
The linux64-clang toolchain alias is currently clang 5. Switch it to
clang 6, but keep the spidermonkey tsan builds on clang 5 because of
bug 1467673.
The LLVM headers that come with clang 6 contain a DEBUG define that
conflicts with our DEBUG define and breaks the clang-plugin build,
so force unset ours.
--HG--
extra : rebase_source : aae88f1166108f003b06c022f14d5f4c61fc1ed9
Chains a release-eme-free-repack-beetmover-checksums kind after release-eme-free-repack-beetmover, to move
the target.checksums generated by the latter into the beetmover-checksums/ in candidates directory. Those
are then consumed by release-generate-checksum kind.
A lot of details like scopes, worker & provisioner, attributes, as well as data like repack_id and
partner_path, are inherited directly from the parent beetmover task. Mainly to avoid recalculating them.
In contrast to nightly builds, GPG signing of target.checksums has not been implemented. I don't believe
that adds any value in our current system because the sigs are not verified.
MozReview-Commit-ID: 38iz3J2PAXh
--HG--
extra : rebase_source : 8f2bebe747d97437780f1bc0d9e2f42159aee8a6
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : source : 0b941cbdca76fb2fbb98dc5bbc1a0237c69954d0
extra : histedit_source : a3e43bdd8a9a58550bef02fec3be832ca304ea93
Let's install python-zstandard for both Python 2 and Python 3 in
all our Debian-based images so it is readily available for use.
MozReview-Commit-ID: 1L8zDc5MYXA
--HG--
extra : rebase_source : db718891dd31d4feceff76fbce753b63049e20b1
python-zstandard's 0.9.1 source distribution contains a debian/
directory.
On Squeeze, producing a Debian package is straightforward.
On Wheezy, we need to hack up Build-Depends because Wheezy doesn't
have a package for the Hypothesis fuzzing library. This package is
only used for testing and our package building disables testing,
so we don't even need to further hack up the packaging to disable
tests.
MozReview-Commit-ID: 6raXjdzggCH
--HG--
extra : rebase_source : 672492a40d65df8430eb17ba033bcb1c0890b7df
dh-python isn't available in Wheezy. Let's backport it so we can
build Python packages that use it.
Fortunately for us, the package builds without any modifications.
The only customization we need is to ensure our custom Python
packages are present in order to satisfy Build-Depends.
MozReview-Commit-ID: CqZtwvosA6K
--HG--
extra : rebase_source : 36515905a6c5937ba16f5f4b566b61715b4f26ac
This build target doesn't have LTO enabled on it (yet)
MozReview-Commit-ID: 56tAHMyvH7o
--HG--
extra : rebase_source : 90039cd8e97332e2ef8aad7908b8a04b2869f4a5
The gecko-{L}-b-macosx64 worker types are really Linux (macOS builds are
cross-compiled). These worker types are essentially identical to their
gecko-{L}-b-linux counterparts.
I don't see a compelling reason to maintain separate worker types for
these builds other than maybe cost accounting (worker types are tagged
in AWS land and these tags can be more easily broken out for billing
analysis). But I don't think any important systems are relying on
this "feature."
So let's move the macOS build tasks to the gecko-{L}-b-linux workers.
MozReview-Commit-ID: 67bArn6IG9T
--HG--
extra : rebase_source : 4de6bf450e7d0d982a770ca8a92e1ac1982fa228
Task run time for these tests is highly variable across chunks: Some run
in only 30 minutes, while xpcshell-11 sometimes exceeds 90 minutes.
Rather than waste resources on more chunks, I think increasing the max
task run time is a reasonable way of avoiding intermittent task time-outs.
While some builds have a PERFHERDER_EXTRA_OPTIONS environment set on the
taskcluster side, many others have the equivalent set at the mozharness
level. But only the former are actually linted against, which,
unsurprisingly, translates to conflicting values between some of the
mozharness configs.
So we move those configurations to taskcluster, enable the lint on all
the kinds that look like builds (based on them using the build_attrs
transform), and adjust the values to stop conflicting. Notably, for
searchfox and static-analysis-autotest.
--HG--
extra : rebase_source : 097333608e61e1df66e5d8f914e15784f35e58f2
This will get the js-bench tasks to run on physical hardware instead of AWS.
MozReview-Commit-ID: 44XavXAwxxn
--HG--
extra : rebase_source : ae1ba4c7f90b3a8526511a3f3c1dff913a334619
I've deliberately left as tier-3 the following tests:
- gtest (perma-OOM, likely from ASan malloc-meddling)
- xpcshell (builds need to be signed plus other failures too)
--HG--
extra : rebase_source : 812bf0de11e91c4e952cb5da9163241bd9386246
Summary:
the blocklist and remote-settings changes need to happen on beta,
but not the hsts/hpkp updates, so we have to split out the control of what
runs by project.
Reviewers: jlorenzo
Reviewed By: jlorenzo
Bug #: 1436369
Differential Revision: https://phabricator.services.mozilla.com/D1487
--HG--
extra : rebase_source : 19ccbb67b880ee7bd2dc2a37325dd70de635abad
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1450
--HG--
extra : rebase_source : b2a6fe48ab031a3d7915bafe30fa8f603ec92d51
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1409
--HG--
extra : rebase_source : 2798c5bc3e3153f8c293846d5a3d786e18bbdc34
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1408
--HG--
extra : rebase_source : 6605d320082c767699e0c360cefa8a04e5525d10
