This adds a 'use-artifacts' key to the run_task schema. Tasks can specify artifacts to download like this:
run:
using: run-task
use_artifacts:
build:
- target.tar.bz2
- target.common.tests.zip
- target.mochitest.tests.zip
This will cause the run-task script to download those three artifacts from the task's 'build' dependency.
If the task doesn't have a 'build' dependency, taskgraph generation will error. The artifacts will be
downloaded into $USE_ARTIFACT_PATH. It is up to the task to do whatever extracting/setup may be required.
E.g this setup could go in the task's command.
At this time, only 'run-task' tasks using docker-worker are supported.
MozReview-Commit-ID: 3f02oCys62i
--HG--
extra : rebase_source : e8a85040e45042b537d4119334c4a8b7280b295c
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : source : 94a9641c5a018cfe729ebe748e75a7c4373e4322
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : source : 4902cab3ce5dab2d1756cf0cd5c95f40603c0a0e
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : source : 19fe5702cf6d018b743108b35e86d1750f205a76
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : source : ef477a048b575958be74287a2273830813b385f1
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : rebase_source : d1450979e636387a5fdbd80ba19a92ec4fd31088
extra : histedit_source : 62910e29cc2fff0a1db0e9521cc58b200406d393
Python 3 is the future.
MozReview-Commit-ID: APuu4Q3mimj
--HG--
extra : rebase_source : 79839dc7f3ec130e467f1d0aa268bf3912cc1c8f
extra : histedit_source : 4be41658b5debaa6165228f60f09a9d1bf53ddbb
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : rebase_source : aa50a6cd1337fe604015131b053234a2db642e57
extra : histedit_source : 8481b23bde4f408a59c623b8be43ee0611492afd
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : rebase_source : 1a45b158fb625c7fd86701736b16da8df122218d
extra : histedit_source : f66fb22e86caf9b6b3cb301bedeab0b709685ef3
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : rebase_source : 074cbf7daaed820bd330b6850d28acd766b3801a
extra : histedit_source : e10c0a14f55148dd3e57a194d262ded41c038ee5
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : rebase_source : bbf3a0bd6cc881002d38c58eef64636c5efa6712
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : rebase_source : 6bda98911cf32ce1bc3d651805b473aff18bf1f9
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : rebase_source : 38d5626574d924c25e3fb4eecf29d379c2678e0d
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : rebase_source : b13d859ffaa2d10795312ff5a0f3ebbd5ca06835
Our normal ubuntu 16.04 test image is suitable for hosting an Android x86
emulator with these minor updates: Install kvm and make sure /dev/kvm
rw permissions are open for everyone. Note that /dev/kvm is generally
only visible when running docker with --privileged; its permissions
cannot be modified in the Dockerfile, only at run-time: run-task is the
first opportunity.
Volumes are a docker-worker concept. They shouldn't be encountered on
Windows, which uses generic-worker.
MozReview-Commit-ID: KUdSxVHVJQ
--HG--
extra : rebase_source : ff54131d471ae2ae2465b11ca5ba6e787f5d11de
main() is quite long. And the control flow will become more complicated
as we support Windows.
Let's move the bulk of the cache configuration code into a standalone
function so main() is less cluttered.
MozReview-Commit-ID: xredCubr1E
--HG--
extra : rebase_source : 385fe6fe9e083cf585edc922b1fa26355d580c02
The code for cache and volume normalization makes assumptions that uid
and gid are defined. They are not defined if not running as root and
Python would crash in these code paths.
So, presumably this means that all tasks using run-task are running as
root. Let's codify that requirement.
This requirement is arbitrary. But let's not scope bloat run-task to
support scenarios until we need them.
MozReview-Commit-ID: 2uW4OSovzWi
--HG--
extra : rebase_source : 56ced8ecbd0eef3a55dc68413f4eab7601756cc0
I want to make run-task work on Windows. The script is currently very
POSIX oriented, as it assumes the existence of the grp and pwd modules,
that user IDs are numeric, and that system calls like setresgid() and
setresuid() are available, etc.
This commit starts to make some of the POSIX-centric code conditional
on running on POSIX.
Code for uid/gid extraction has been moved to its own function. Some
error messages were tweaked slightly as part of the move. Otherwise,
the changes should be pretty straightforward.
There are still other parts of this file that won't work on e.g.
Windows. But this gets us a big step closer.
MozReview-Commit-ID: HNyytKcBbBo
--HG--
extra : rebase_source : 01d653c801865e36e562a8de4c9e6d6962498f19
In preparation for making it usable on Windows, after which point
having it in a directory with "docker" in it doesn't make much sense.
MozReview-Commit-ID: Hgu0buFyJwF
--HG--
rename : taskcluster/docker/recipes/run-task => taskcluster/scripts/run-task
extra : rebase_source : 3c0b502d28b5aad54bd04069efbfda88e25bbb20
This option shouldn't be used for local builds (see bug 1294157). Set
the option from the crate's taskcluster script instead, so that it's
used only for automated builds.
--HG--
extra : rebase_source : 94b398a0f1fac60094269d755735c426ecc63e17
This will allow mozharness configs to be specified exclusively in the taskgraph.
Differential Revision: https://phabricator.services.mozilla.com/D1017
--HG--
extra : rebase_source : a3a9b6cc9d1004c4bd396fccc3e4354a7316651d
extra : source : 10acd193df92b7c495789dc24157b85f116ade5e
And adapt the build-gcc.sh script for the changes to
contrib/download_prerequisites.
--HG--
rename : taskcluster/scripts/misc/build-gcc-6-linux.sh => taskcluster/scripts/misc/build-gcc-7-linux.sh
extra : rebase_source : b1d785777b8c141c0eb0f52a73734abd2db21b94
The URL is now being redirected to
https://www.openssl.org/source/old/1.1.0/openssl-1.1.0g.tar.gz. Let's
add a -L so we follow redirects automatically.
MozReview-Commit-ID: AuZ98jGidzl
--HG--
extra : rebase_source : 07e61558024e789df45d8e2ab67ab5ad9d3d355b
Build the latest tup master branch with the LD_PRELOAD dependency
checker.
MozReview-Commit-ID: ALfnnmOZrky
--HG--
extra : rebase_source : 529d4392ef73e03f66fb76f089f8b88f45b44972
Note that static analysis was the only remaining user of the 32-bit toolchain, so I've removed win32-clang-cl (or rather, renamed it to win32-clang-cl-st-an).
--HG--
rename : build/build-clang/clang-win32.json => build/build-clang/clang-win32-st-an.json
rename : build/build-clang/clang-win64.json => build/build-clang/clang-win64-st-an.json
rename : taskcluster/scripts/misc/build-clang32-windows.sh => taskcluster/scripts/misc/build-clang32-st-an-windows.sh
rename : taskcluster/scripts/misc/build-clang64-windows.sh => taskcluster/scripts/misc/build-clang64-st-an-windows.sh
Ensure better determinism when creating rust toolchain packages
by rejecting generic channels like 'stable' or 'nightly'. Instead,
insist on a specific version or date.
The current valid dates for beta and nightly can be obtained with:
curl -s https://static.rust-lang.org/dist/channel-rust-beta.toml | grep ^date
curl -s https://static.rust-lang.org/dist/channel-rust-nightly.toml | grep ^date
MozReview-Commit-ID: I0DXw1KJGZz
--HG--
extra : rebase_source : 92e158193072582b8568d9c9f00ffdefa0af1a9c
The Proguard dependency is now managed by Gradle.
MozReview-Commit-ID: EOvKSE5z28P
--HG--
extra : rebase_source : 760b117f500cc639cc8c24e9c02933990f358dd7
We'd like to install the NDK through the Android SDK manager. But we
can't pin versions of the NDK with the SDK manager, and so Google
can silently upgrade the NDK on us. Since that is undesirable, this is
the next best thing.
With the toolchain task in hand, we can make all the relevant tasks
depend on the toolchain task and remove the download of the NDK from
tooltool as well.
