1. Updated hgrepo to work with mozilla-beta, mozilla-esr60 and project branches (just in case)
2. Presquashed commits, so we only submit one.
3. Replaced 'which' with 'command -v' to avoid future shellcheck issues.
Differential Revision: https://phabricator.services.mozilla.com/D1582
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : rebase_source : 4918b8c3bac53d63665006802054038bfbca0314
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : source : 0b941cbdca76fb2fbb98dc5bbc1a0237c69954d0
extra : histedit_source : a3e43bdd8a9a58550bef02fec3be832ca304ea93
After this change, we consistently import GPG keys from files in
the GCC build scripts.
MozReview-Commit-ID: BcyvCQoGbMS
--HG--
extra : rebase_source : 657ccce8e242cabdfaff396fd0d6439754a3f364
After this change, we consistently import GPG keys from files in
the GCC build scripts.
MozReview-Commit-ID: BcyvCQoGbMS
--HG--
extra : source : 5fce34a460b51e45ac280a9f0cb8bad896fbcff1
extra : histedit_source : 01621ea8111315c251a9493a11efca72c2ba3c7d
Let's install python-zstandard for both Python 2 and Python 3 in
all our Debian-based images so it is readily available for use.
MozReview-Commit-ID: 1L8zDc5MYXA
--HG--
extra : rebase_source : db718891dd31d4feceff76fbce753b63049e20b1
python-zstandard's 0.9.1 source distribution contains a debian/
directory.
On Squeeze, producing a Debian package is straightforward.
On Wheezy, we need to hack up Build-Depends because Wheezy doesn't
have a package for the Hypothesis fuzzing library. This package is
only used for testing and our package building disables testing,
so we don't even need to further hack up the packaging to disable
tests.
MozReview-Commit-ID: 6raXjdzggCH
--HG--
extra : rebase_source : 672492a40d65df8430eb17ba033bcb1c0890b7df
dh-python isn't available in Wheezy. Let's backport it so we can
build Python packages that use it.
Fortunately for us, the package builds without any modifications.
The only customization we need is to ensure our custom Python
packages are present in order to satisfy Build-Depends.
MozReview-Commit-ID: CqZtwvosA6K
--HG--
extra : rebase_source : 36515905a6c5937ba16f5f4b566b61715b4f26ac
This build target doesn't have LTO enabled on it (yet)
MozReview-Commit-ID: 56tAHMyvH7o
--HG--
extra : rebase_source : 90039cd8e97332e2ef8aad7908b8a04b2869f4a5
They are no longer used. By not listing them, we prevent new tasks
from using them.
MozReview-Commit-ID: FiYhV8WcAsm
--HG--
extra : rebase_source : 18a5be1c050b622f1c57f752b3f97563009cdaf1
For the same reasons that we moved build tasks to this worker type.
MozReview-Commit-ID: LZuzDtHSKL6
--HG--
extra : rebase_source : 6fa455cde001f966024186a26e0589fc9f139293
The gecko-{L}-b-macosx64 worker types are really Linux (macOS builds are
cross-compiled). These worker types are essentially identical to their
gecko-{L}-b-linux counterparts.
I don't see a compelling reason to maintain separate worker types for
these builds other than maybe cost accounting (worker types are tagged
in AWS land and these tags can be more easily broken out for billing
analysis). But I don't think any important systems are relying on
this "feature."
So let's move the macOS build tasks to the gecko-{L}-b-linux workers.
MozReview-Commit-ID: 67bArn6IG9T
--HG--
extra : rebase_source : 4de6bf450e7d0d982a770ca8a92e1ac1982fa228
Task run time for these tests is highly variable across chunks: Some run
in only 30 minutes, while xpcshell-11 sometimes exceeds 90 minutes.
Rather than waste resources on more chunks, I think increasing the max
task run time is a reasonable way of avoiding intermittent task time-outs.
While some builds have a PERFHERDER_EXTRA_OPTIONS environment set on the
taskcluster side, many others have the equivalent set at the mozharness
level. But only the former are actually linted against, which,
unsurprisingly, translates to conflicting values between some of the
mozharness configs.
So we move those configurations to taskcluster, enable the lint on all
the kinds that look like builds (based on them using the build_attrs
transform), and adjust the values to stop conflicting. Notably, for
searchfox and static-analysis-autotest.
--HG--
extra : rebase_source : 097333608e61e1df66e5d8f914e15784f35e58f2
This will get the js-bench tasks to run on physical hardware instead of AWS.
MozReview-Commit-ID: 44XavXAwxxn
--HG--
extra : rebase_source : ae1ba4c7f90b3a8526511a3f3c1dff913a334619
There is a superficial check in the run-task script which requires root. Simply
removing this check allows a native-engine task (which isn't running as root)
to proceed.
MozReview-Commit-ID: 44XavXAwxxn
--HG--
extra : rebase_source : bd1f01ce1c2feb4029838e07314493d449a4f46e
This adds an optional 'workdir' key to all job schemas. It still defaults to
/builds/worker, but can be overriden by individual tasks or schema
implementations.
MozReview-Commit-ID: LY20xfBhbCP
--HG--
extra : rebase_source : 7ac76ebf55d33d30c2aad73484421c6b4002cd33
Extends support of the use-artifacts key to native-engine based tasks.
MozReview-Commit-ID: FJILoyD5XVZ
--HG--
extra : rebase_source : 0cf8bf63f73d0fbb634f6b437bcc9bcce7821900
I've deliberately left as tier-3 the following tests:
- gtest (perma-OOM, likely from ASan malloc-meddling)
- xpcshell (builds need to be signed plus other failures too)
--HG--
extra : rebase_source : 812bf0de11e91c4e952cb5da9163241bd9386246
Summary:
the blocklist and remote-settings changes need to happen on beta,
but not the hsts/hpkp updates, so we have to split out the control of what
runs by project.
Reviewers: jlorenzo
Reviewed By: jlorenzo
Bug #: 1436369
Differential Revision: https://phabricator.services.mozilla.com/D1487
--HG--
extra : rebase_source : 19ccbb67b880ee7bd2dc2a37325dd70de635abad
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1450
--HG--
extra : rebase_source : b2a6fe48ab031a3d7915bafe30fa8f603ec92d51
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1449
--HG--
extra : rebase_source : f63e6c5a13904ba33fa2a46e05cfdd0997abd0fc
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1409
--HG--
extra : rebase_source : 2798c5bc3e3153f8c293846d5a3d786e18bbdc34
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1408
--HG--
extra : rebase_source : 6605d320082c767699e0c360cefa8a04e5525d10
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1406
--HG--
extra : rebase_source : 207d1c25e37ab2619a09fb209282ffe55025de26
The crash reporter symbol files are the easiest cross-platform way to
find static initializers. While some types of static initializers (e.g.
__attribute__(constructor) functions) don't appear there in a notable
way, the static initializers we do care the most about for tracking do
(static initializers from C++ globals). As a matter of fact, there is
only a difference of 2 compared to the currently reported count of 125
on a linux64 build, so this is a good enough approximation. And allows
us to easily track the count on Android, OSX and Windows builds, which
we currently don't do.
The tricky part is that the symbol files are in
dist/crashreporter-symbols/$lib/$fileid/$lib.sym, and $fileid is hard to
figure out. There is a `fileid` tool in testing/tools, but it is a
target binary, meaning it's not available on cross builds (OSX,
Android).
So the simplest is just to gather the data while creating the symbol
files, which unfortunately requires to go through some hoops to make it
happen for just the files we care about.
--HG--
extra : rebase_source : 458fed1ffd6f9294eefef61f10ff7a284af0d986
Thunderbird releases need to look at comm-beta/comm-esr* branches for old
locale/version information.
Differential Revision: https://phabricator.services.mozilla.com/D1413
--HG--
extra : rebase_source : 76625ea5859d25f270b9fbec577f9075988bf2b7
PyYAML is not fast, so the fewer times we parse the same file, the better
MozReview-Commit-ID: KuYKFY7hFXp
--HG--
extra : rebase_source : 55df7c515db8864ee6d01895d444f7f26229bc2f
When running jobs on try, we shouldn't have access to the partials, so
don't try to give them.
Differential Revision: https://phabricator.services.mozilla.com/D1352
--HG--
extra : source : 90d19b52f956ce509525ca160ef5a2af80eea1a9
extra : histedit_source : b303ac9a4117a4e8e047d6459a7ba6477b74b762
For kind=hook, the spec doesn't include this value as it's untrustworthy.
For kind=task, it's still untrustworthy, but there is no privilege escalation
so that's not important. Still, it dramatically expands the size of the task
definition.
MozReview-Commit-ID: 6scQ2ZwxP10
--HG--
extra : rebase_source : 4dc34390a510091ddc26023755992995fe358e47
In a post-actions-as-hooks world, users will not have scopes to create tasks,
so this mode of action definition will not be possible. This is not currently
used from Treeherder (it links to
https://tools.taskcluster.net/tasks/<taskid>/interactive instead)
This drops support for the JSON-e-only interactive action; that action is not
currently used from treeherder, so that should have no impact for users.
MozReview-Commit-ID: 9i3POpjahAc
--HG--
extra : rebase_source : e6de03389a0c5c67d5332d2b1c97e1d4bf6a22d3
support_vcs_checkout() always sets the environment variables that
were set by this deleted function. In addition, support_vcs_checkout()
also adds caches and scopes - at least for docker-worker and
docker-engine. For generic-worker - which was used in all call sites
of docker_worker_add_gecko_vcs_env_vars() (yes, the "docker_worker"
bit of the name was completely wrong - probably a legacy holdover) -
support_vcs_checkout() was *almost* exactly equivalent to
docker_worker_add_gecko_vcs_env_vars(). The only difference is that
support_vcs_checkout() adds the
secrets:get:project/taskcluster/gecko/hgfingerprint scope in addition
to setting the environment variables.
MozReview-Commit-ID: 8fl3u9be5fT
--HG--
extra : rebase_source : 0eec2f143f903a3fcc5502b60026f5d8061100ea
This functionality was implemented at least 3 times. Let's consolidate
it to a central function.
Returning multiple command strings is kind of funky. I preserved
existing behavior and mozharness jobs are the only ones printing the
forensic logging. We should probably move this logging into
robustcheckout so we don't need to involve taskgraph with this. But
that can be deferred to another day.
MozReview-Commit-ID: I2LglJvfI6
--HG--
extra : rebase_source : 7cb413694aee4e46a6522febe9daa4b73b5307ca
extra : source : 096d7d374af427ee950c7a550878781eebad4135
The geckoview-junit tests require the OSS audio backend for the Android
4.3 ARM emulator, but mochitests don't work well with the OSS audio
backend. Therefore, use a different config file for the geckoview-junit
tests.
MozReview-Commit-ID: 20tzjtVdTuB
The email address used for notification on try was templated,
but nothing actually evaluated the template. This applies the same
templating that applies to the message to any emails specified.
Differential Revision: https://phabricator.services.mozilla.com/D1297
--HG--
extra : source : dd32a78ddf2196436f2098b4bc8bd3dc5c77b526
extra : amend_source : 8eac858e2b658bb2d8c3dacabe4c7fa3c077d9bc
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : source : 94a9641c5a018cfe729ebe748e75a7c4373e4322
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : source : 4902cab3ce5dab2d1756cf0cd5c95f40603c0a0e
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : source : 19fe5702cf6d018b743108b35e86d1750f205a76
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : source : ef477a048b575958be74287a2273830813b385f1
Our normal ubuntu 16.04 test image is suitable for hosting an Android x86
emulator with these minor updates: Install kvm and make sure /dev/kvm
rw permissions are open for everyone. Note that /dev/kvm is generally
only visible when running docker with --privileged; its permissions
cannot be modified in the Dockerfile, only at run-time: run-task is the
first opportunity.
download-and-compress isn't very complicated and should work on Python 3
with minimal effort. So let's switch it to use Python 3.
MozReview-Commit-ID: 9G1WfcbbKEY
--HG--
extra : rebase_source : 3a6bab06c8500a90413e8b7642a7bf7bdff04a46
python-zstandard 0.9 has an API that exposes a file object interface
for compression and decompression. This means we can remove our
stream wrapper in order to consume a zstandard compressed tar file.
MozReview-Commit-ID: DeWWKnigJVa
--HG--
extra : rebase_source : b510b9c7cf4471df835c755299a7842d13188b67
The latest python-zstandard uses a newer zstandard that is faster.
It also has wheels available, which means installation doesn't require
Python development headers, etc.
MozReview-Commit-ID: 5gRq81KYmX4
--HG--
extra : rebase_source : 96ccc64e9707c6b4815c1bfa5c1a98b9a428b387
Version 0.9.0 bundles a newer version of the zstandard library, which
is a little faster and has a few minor bug fixes (none that we were
likely hitting, however).
MozReview-Commit-ID: 9YgSZ0G41eg
--HG--
extra : rebase_source : 8f5a68323b1e1fe7e9f1dd1a92e132434972d21d
We want Python 3 available everywhere because it is 2018.
MozReview-Commit-ID: L3wufNXKdnp
--HG--
extra : rebase_source : c260923e3c13f8b28e30eaaf6e1bd38f79500052
Address Sanitizer builds use --disable-crashreporter so they don't have
symbols zip files to upload. Don't run symbol upload tasks for these builds.
MozReview-Commit-ID: GeQgRZF3m8t
--HG--
extra : rebase_source : 6ac59c5c96b5fb5ddbbe8c60af3a203d02ea9883
Mainly so searching "toolchain" + "gcc" yields something useful in the
taskgraph.
MozReview-Commit-ID: HWiT3AwwYQ2
--HG--
extra : rebase_source : b1ba0dfb4f99b6f8abe42506e8b37db68ed03590
Instead of having a special test-set for linux64-qr we can just use the
regular test set, and explicitly disable the individual tests that are
failing.
MozReview-Commit-ID: 8MUj1YdtOsH
--HG--
extra : rebase_source : 5b4398ccedd208c97fe2c58024d98bfdb759c932
Instead of having a special test-set for linux64-qr we can just use the
regular test set, and explicitly disable the individual tests that are
failing.
MozReview-Commit-ID: 8MUj1YdtOsH
--HG--
extra : rebase_source : 4d38dd3ea7a6934c84e57d6a20c7dc457f06c2da
Note in particular that tasks that were previously set to run on just
['mozilla-central', 'try'] will now also run on inbound and autoland, in
addition to mozilla-beta and other release branches. In some cases (e.g.
for talos tests) this might result in a significant increase in load on
CI infrastructure. For the tasks that were already running on ['trunk',
'try'] the extra load from the release branches should be relatively
small and will only take effect once 62 moves off nightly into beta.
MozReview-Commit-ID: 6sn9q6rCxOK
--HG--
extra : rebase_source : c228adf059a950aec3e311ae11915caf345e854f
Mainly so searching "toolchain" + "gcc" yields something useful in the
taskgraph.
MozReview-Commit-ID: HWiT3AwwYQ2
--HG--
extra : rebase_source : b1ba0dfb4f99b6f8abe42506e8b37db68ed03590
Summary:
In essence, we're allowing a new field in the task definition,
which is trusted over the one that's passed in with the config. This
wouldn't make much sense if it had a real date in, but allows us to
set an empty string for the kind that needs it
Reviewers: bhearsum
Reviewed By: bhearsum
Bug #: 1458854
Differential Revision: https://phabricator.services.mozilla.com/D1214
--HG--
extra : rebase_source : 82145a94fa91957ffe57112a1c0d327d99e32b23
This Dockerfile downloads non-deterministic remote content (by cloning a
Git repo) and then executes code from it. Part of that code is
executing Python package installs.
Since this Docker image was generated, it appears the remote code
requires new build dependencies. This commit adds those package
dependencies.
Not having deterministic Docker image builds is a bug. I'll file a
follow-up so we pin the Git commit used for building so this type
of failure doesn't occur again.
--HG--
extra : amend_source : 533a95abeb7cf7ddc9a1329549f5d294baf983f5
Volumes are a docker-worker concept. They shouldn't be encountered on
Windows, which uses generic-worker.
MozReview-Commit-ID: KUdSxVHVJQ
--HG--
extra : rebase_source : ff54131d471ae2ae2465b11ca5ba6e787f5d11de
main() is quite long. And the control flow will become more complicated
as we support Windows.
Let's move the bulk of the cache configuration code into a standalone
function so main() is less cluttered.
MozReview-Commit-ID: xredCubr1E
--HG--
extra : rebase_source : 385fe6fe9e083cf585edc922b1fa26355d580c02
The code for cache and volume normalization makes assumptions that uid
and gid are defined. They are not defined if not running as root and
Python would crash in these code paths.
So, presumably this means that all tasks using run-task are running as
root. Let's codify that requirement.
This requirement is arbitrary. But let's not scope bloat run-task to
support scenarios until we need them.
MozReview-Commit-ID: 2uW4OSovzWi
--HG--
extra : rebase_source : 56ced8ecbd0eef3a55dc68413f4eab7601756cc0
I want to make run-task work on Windows. The script is currently very
POSIX oriented, as it assumes the existence of the grp and pwd modules,
that user IDs are numeric, and that system calls like setresgid() and
setresuid() are available, etc.
This commit starts to make some of the POSIX-centric code conditional
on running on POSIX.
Code for uid/gid extraction has been moved to its own function. Some
error messages were tweaked slightly as part of the move. Otherwise,
the changes should be pretty straightforward.
There are still other parts of this file that won't work on e.g.
Windows. But this gets us a big step closer.
MozReview-Commit-ID: HNyytKcBbBo
--HG--
extra : rebase_source : 01d653c801865e36e562a8de4c9e6d6962498f19
In preparation for making it usable on Windows, after which point
having it in a directory with "docker" in it doesn't make much sense.
MozReview-Commit-ID: Hgu0buFyJwF
--HG--
rename : taskcluster/docker/recipes/run-task => taskcluster/scripts/run-task
extra : rebase_source : 3c0b502d28b5aad54bd04069efbfda88e25bbb20
Summary: Now with correct phabricator usernames
Reviewers: aki
Reviewed By: aki
Bug #: 1458855
Differential Revision: https://phabricator.services.mozilla.com/D1133
--HG--
extra : rebase_source : fd8aa5de1c1aaedacbbad53fa692e6c3132a0a94
The python3-minimal package provides /usr/bin/python3 on Debian.
This commit installs this package so a `python3` executable is
provided.
This required backporting the package to wheezy. The final patch
is trivial. But I wasted a bit of time figuring out why `mk-build-deps`
wasn't working. It would no-op and exit 0 and then the build would
complain about missing dependencies!
glandium's theory is that the ":any" multiarch support on wheezy
isn't complete. Removing ":any" seems to make things "just work."
MozReview-Commit-ID: FBicpK4SmkQ
--HG--
extra : rebase_source : a28ce731024e8ed6a43fb30e2ed57da2abb50d0f
If we're not passed `release_partner_config` in the input, let's poll github for it in the action.
MozReview-Commit-ID: 2swx76vhTE5
--HG--
extra : rebase_source : d16c517348e506519bc6e6296ad7a0cb2f90527c
extra : source : 3ad2e53b051d48a54dfb1cf88b743ed4d9571ff6
We originally had this logic here, and called it from the `partner_repack` transform. This kept the config more hidden, but had the downsides of a) being difficult to test, and b) hitting the network during transforms, which we're trying to avoid.
We moved this code to release-runner3, and passed in the `release_partner_config` as input, and saved it as a parameter. Parameterizing the partner config means that we can refer to it easily throughout taskgraph generation and in local testing, and we don't have to hit the network during transforms. The downsides include potentially having to generate this config in multiple places (rr3, ship-it-v2, the partner hook), and risking hitting the 20k gpg cleartext character limit in the `ACTION_INPUT` environment variable.
Now I'm moving this code back into util.partners, but I'm calling it from the action, not from the transform. The action populates the `release_partner_config` parameter, so we can still access the config from anywhere in the taskgraph generation code, more easily test, and avoid hitting the network during transforms. It also means that release-runner3, ship-it-v2, and the partner hook can all use the partner config generation code from a single location, rather than having to duplicate it.
Hoping this is the last major change we need to make here for a while.
MozReview-Commit-ID: 8UmvmcAoZgD
--HG--
extra : rebase_source : 57464354ea294bec972d6ff44b8c9426b31fc644
extra : source : 21289762391e66c9330f5c6f93a417f8d608c6e9
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : rebase_source : d1450979e636387a5fdbd80ba19a92ec4fd31088
extra : histedit_source : 62910e29cc2fff0a1db0e9521cc58b200406d393
Python 3 is the future.
MozReview-Commit-ID: APuu4Q3mimj
--HG--
extra : rebase_source : 79839dc7f3ec130e467f1d0aa268bf3912cc1c8f
extra : histedit_source : 4be41658b5debaa6165228f60f09a9d1bf53ddbb
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : rebase_source : aa50a6cd1337fe604015131b053234a2db642e57
extra : histedit_source : 8481b23bde4f408a59c623b8be43ee0611492afd
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : rebase_source : 1a45b158fb625c7fd86701736b16da8df122218d
extra : histedit_source : f66fb22e86caf9b6b3cb301bedeab0b709685ef3
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : rebase_source : 074cbf7daaed820bd330b6850d28acd766b3801a
extra : histedit_source : e10c0a14f55148dd3e57a194d262ded41c038ee5
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : rebase_source : bbf3a0bd6cc881002d38c58eef64636c5efa6712
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : rebase_source : 6bda98911cf32ce1bc3d651805b473aff18bf1f9
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : rebase_source : 38d5626574d924c25e3fb4eecf29d379c2678e0d
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : rebase_source : b13d859ffaa2d10795312ff5a0f3ebbd5ca06835
Right now artifacts from previous tasks are left lying around. We should clean these up
in case a task accidentally uses an artifact from the wrong dependency.
Ideally we'd cache these properly based on the taskId they came from, but that can be
follow-up fodder.
MozReview-Commit-ID: HUgvNlqyFav
--HG--
extra : rebase_source : fb9c6723598223619993c2695fb588ead3325edb
Previously we would only generate upload-symbols tasks for nightly builds,
since we only want to upload symbols for builds we ship to users.
On try, developers may want to use the symbol server but very rarely want
to do nightly builds, so allow uploading symbols from any build type there.
MozReview-Commit-ID: IYs9mZii3DN
--HG--
extra : rebase_source : 15acb889510bb07ed3d29ea802997f11796dad9f
Tasks like upload-symbols that have a dependency on a build job want to
copy the treeherder.platform from the build job but it gets lost in
the task transform currently. This simply copies it into
an extra.treeherder-platform key to make life easier.
MozReview-Commit-ID: H4PtC4mvIYA
--HG--
extra : rebase_source : 5588bdadfcff8eb8f35935dc5c05dcde5658da83
filter_upload_symbols is a relic of task configurations that were written
before we had a better handle on taskgraph generation. We should only be
uploading symbols for nightly builds anyway, so this is better served using
newer filtering methods.
upload-symbols tasks were specified to run on non-nightly build types in the
kind.yml, but those were filtered out in filter_upload_symbols. I believe
these were simply an artifact of the initial upload-symbols implementation
happening before nightly builds were running in Taskcluster.
MozReview-Commit-ID: Je1NytrVPT8
--HG--
extra : rebase_source : a961c17d329af848fa7bb64c5186135d37dd412f
This is a follow up to 9c941daebe9fb3e79066ee4def16ed5ce0eb10a9
Differential Revision: https://phabricator.services.mozilla.com/D1140
--HG--
extra : rebase_source : 4c56ea683a7bcb3a80cd032e5491d3c13e891368
extra : source : 72e960bcc54751b15d144e0a5550c65649a2ad7d
