This is simply diagnostic; it's nice to see what the versions of the
Android packages installed are, like:
Installed packages:
Path | Version | Description | Location
------- | ------- | ------- | -------
<snip>
platforms;android-23 | 3 | Android SDK Platform 23 | platforms/android-23/
tools | 26.0.1 | Android SDK Tools 26.0.1 | tools/
This is really useful because it's not possible to pin most packages
to specific versions; that is, we can only install latest "tools", not
"tools;26.0.1".
MozReview-Commit-ID: HgZLGCAObEs
--HG--
extra : rebase_source : 33e5c9f81d05551c9e167eac62009f1de023b410
This adds a new `tc-A` Tree Herder group, similar to the `A` Autophone
(or Android) group. (I don't want to include a `g` for Gradle because
eventually there will be nothing that is _not_ Gradle.)
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1371445#c31, the
sheriffs are satisfied with the test output.
As to the rest of
https://bugzilla.mozilla.org/show_bug.cgi?id=1372075#c0, the
documentation is in place, and |mach try fuzzy| has eclipsed
trychooser, so I think we should not update trychooser.
MozReview-Commit-ID: 2OWDEmGCd11
--HG--
extra : rebase_source : 8c62f958b64c38797e9070e8328cd7f24baa8cc5
The multi-l10n and update mozharness actions are Nightly build
specific. The android-* tasks look like builds but aren't really
builds -- they don't produce APKs, for example. In the past, these
actions immediately returned because the android-* jobs where never
considered Nightly builds and there is a test for "is Nightly" in each
mozharness action. When forcing a build to be a Nightly, these
actions don't immediately return; since they're build specific, this
causes errors.
This commit simply doesn't run these actions, since they're not
appropriate to these tasks.
MozReview-Commit-ID: deJJbBu0eb
--HG--
extra : rebase_source : 8a6b9d69bf7a1cbeb30f84d6e5def41c1af3816c
AFAICT, the last use of the build.sh script was removed in bug 1309593,
and checkout-sources.sh is only used from that script.
--HG--
extra : rebase_source : f89d9071af0e49a168e1eb7c38e291c29ed7119f
Except ASAN builds, which for some reason fail with that version
(bug 1409267).
--HG--
extra : rebase_source : e91bd0f4cd152be57abd5cddb8e15e4af34912bb
I don't think (the output of) this script is used anywhere.
MozReview-Commit-ID: DwMFtpozjNL
--HG--
extra : rebase_source : 005086039f520ba116534ab47ee49616c6958e85
The only tricky piece here is that the resulting toolchain archive is
private, and uses a newly allocated Task Cluster scope
(queue:get-artifact:project/gecko/android-sdk/*) to restrict access to
the archive. All SCM levels (1, 2, 3) have been given the new scope:
see https://tools.taskcluster.net/auth/roles/moz-tree:level:1 and
friends.
MozReview-Commit-ID: CcDqDOHODpe
--HG--
extra : rebase_source : 81dbb065f2a3c4e7733e964be66adb1733db52c6
Not all Docker images are configured for tc-vcs caches: in particular,
android-build is not configured. Until we fully remove tc-vcs, this
will let toolchain tasks use non-tc-vcs caching images.
MozReview-Commit-ID: CYSdn2kpF3S
--HG--
extra : rebase_source : 87dec4fdb32290ef396cda7f92da6bd688bc5b7b
I *think* the modifications to MockedOpen are correct, but I'm not sure..
MozReview-Commit-ID: 6vTZBtdQ1dz
--HG--
extra : rebase_source : 2d2008f87640747ef831d000bf13a4b1b7fcd338
Specifying anything but "toolchain-build" appears to have cache
errors, which I cannot understand. These cache errors can be
addressed in follow-up, if and when toolchain tasks require new sparse
profiles. But specifying `null` avoids the sparse profile details
entirely, which works nicely for `android-gradle-dependencies`, which
is build-like and requires a fairly complete checkout.
MozReview-Commit-ID: L3R8UIDjgqW
--HG--
extra : rebase_source : 096a180238d1b5eeffbb0e2d9d538def162e877f
I don't think (the output of) this script is used anywhere.
MozReview-Commit-ID: DwMFtpozjNL
--HG--
extra : rebase_source : 36b3cbe1a6a9e5cd163782c1c13653be8558a03a
The only tricky piece here is that the resulting toolchain archive is
private, and uses a newly allocated Task Cluster scope
(queue:get-artifact:project/gecko/android-sdk/*) to restrict access to
the archive. All SCM levels (1, 2, 3) have been given the new scope:
see https://tools.taskcluster.net/auth/roles/moz-tree:level:1 and
friends.
MozReview-Commit-ID: CcDqDOHODpe
--HG--
extra : rebase_source : 062bca8c65556f0f46e9c9cc6cd81eb04cf2b522
Not all Docker images are configured for tc-vcs caches: in particular,
android-build is not configured. Until we fully remove tc-vcs, this
will let toolchain tasks use non-tc-vcs caching images.
MozReview-Commit-ID: CYSdn2kpF3S
--HG--
extra : rebase_source : 4f032650baaa49537ffd894b34e936af2141a330
The goal of this approach is to tell Gradle to not connect (or allow
it to connect) to the network when fetching dependencies. No Android
automation tasks should fetch from the network, except the toolchain
tasks (which are specially intended to do so).
It's difficult to arrange this without including the `--offline` flag
everywhere. It _should_ be possible to set offline using an
environment variable -- which would allow us to get rid of these
dotgradle-* files -- but offline isn't an option in
https://docs.gradle.org/4.2.1/userguide/build_environment.html#sec:gradle_configuration_properties
(and certainly not in earlier versions either). Therefore,
environment variable that points to an init.gradle file in automation.
Before this patch, the files telling Gradle whether to start offline
were fetched from tooltool. That's just a layer that doesn't need to
be there.
None of this impacts local developers.
MozReview-Commit-ID: LAXktbBu1Az
--HG--
extra : rebase_source : d23801643d32135a87d410bf5e8508da556ef9be
This is just one flavor of the "reftets" suite, so we need to add a distinct
scheduling component for it.
MozReview-Commit-ID: AtKuvuUCk1l
--HG--
extra : rebase_source : 3f316f0293e8d1245fc6e891bbcd044586ab6c06
This approach allows to specify the `docker-image` and set of
`toolchains` to the l10n leaf jobs using the `by-platform:` override
mechanism. We don't support anything but in-tree docker images at
this time, and the schema will warn if a different type of docker
configuration block is used. It wouldn't be hard to grow the
additional blocks, but let's reduce duplication for now.
It might be considered better to inherit the `docker-image` and set of
`toolchains` from the underlying `dependent-task`, but we don't do
that for two reasons. The main reason is that it's an explicit goal
to be able to "cross repack": to repack, say, a Windows binary on a
Linux worker. In that situation, the docker-image and toolchains
differ between the builder and the repack worker.
A smaller technical obstruction is that by the time the l10n transform
sees the dependent task, the docker image and set of toolchains have
been processed. The l10n transform would have to "reconstitute" the
docker image changes and the set of toolchains; it would be very
fragile.
Taken together, it's better to be explicit, reduce unexpected
interactions, and repeat the information in the l10n leaf tasks.
MozReview-Commit-ID: TmgJyYU5dx
--HG--
extra : rebase_source : 9aae494165d9a7c70de0f5fe4849ec219e28a20c
This is a work-around until Bug 1405889 is deployed. Using the /bewit
endpoint does have the advantage of avoiding another issue in
taskcluster-proxy, namely that the /bewit approach streams. Fetching
through the proxy does not stream from the upstream resource; the
upstream resource is fetched and stored in taskcluster-proxy's memory,
increasing operational costs.
MozReview-Commit-ID: 8yS7zKLALhd
--HG--
extra : rebase_source : 23e1bc683248f69f6e4c90204e9bc0701f4a778a
There's code that carefully uses `setdefault('artifacts', [])` in the
same file, but then stomps on 'artifacts' before that's invoked. This
allows tasks to change where public/build is sourced from, and to add
additional artifact locations (including private locations).
MozReview-Commit-ID: JqyHew5bGv5
--HG--
extra : rebase_source : 420f1e062583d315faa413181b1ac17c0e55249e
With this in place, all `-j`obs will not run by default on try. This will omit
such jobs in most try pushes even if files-changed matches. This is
unfortunate, but better than running them unconditionally. Fuzzy selections,
and later `just try it` pushes, are the ultimate solution here.
With this change, a push with no try syntax or try_task_config.json will schedule
no tasks at all.
MozReview-Commit-ID: FGjqlDW1FT6
--HG--
extra : rebase_source : 727ceafb1b6d24f83c0c7382b6a877ecb65863ab
Specifically, this avoids setting optimize_target_tasks to True unconditionally
for non-try branches.
MozReview-Commit-ID: HSJFLmqbMmZ
--HG--
extra : rebase_source : 459e65c0d423d091846134da011b0e201ff8da99
When adding sccache toolchain jobs in bug 1381772, building with gcc
failed, and building with clang worked, so I just went with the path of
least resistance. That's however a suboptimal position in the dependency
graph, so it's still preferable to use gcc if possible.
Looking exactly how it fails, it turns out it's because without CC being
set, ring wants to build with "cc", which ends up being the system gcc
instead of ours (our gcc archive doesn't provide "cc", only "gcc"), and
it is too old to support the compiler flags ring uses.
So setting CC does the trick.
--HG--
extra : rebase_source : 4c657664957dff1f7aebe470e0440a52c9e280e5
New upstream stable release.
Maintain rust-1.19 builds for verifying the minimum-supported
version, and because we have network failures when we build
sccache with rust 1.20 or 1.21.
MozReview-Commit-ID: 5qi8JDTjfzj
By the schedule we would have bumped the minimum-supported rust
version to 1.20.0 some weeks ago, just before Firefox 57 went to
beta. However, that was blocked on a couple of compatibility
issues with sccache and dsymutil. See bug 1396884.
Now that 1.20.0 is working in our automation builds, require
it for all builds to unblock servo and webrender upgrades.
Also update our integration test to verify building with
the new minimum.
MozReview-Commit-ID: 8otdix6f45f
--HG--
extra : rebase_source : 7d2550e8ddc772b45602bd488f174fc180e91484
for: Run buildbot's periodic_file_update job scheduled via taskcluster.
I messed up thinking this was filter-out not filter in the target task method.
I'm also renaming the target_task method in order to avoid these decision jobs
from needing to contact balrog for partial data (because it had 'nightly' in the
target task name.
MozReview-Commit-ID: 3uetnWG4vnW
--HG--
extra : rebase_source : 82dc838d23e02ae2ea515416a29bb0b491c053b9
At this point, we could tear out the `ignore-locales` attribute, since
l10n-bumper supplies that information. However, we may want to use flatfiles
for something; `ignore-locales` allows for that.
MozReview-Commit-ID: 8mD4iav3bKx
--HG--
extra : rebase_source : abe2075503838223a2c150676b9c72a1aa74df59
Diffing `target-graph`s was difficult because the locales kept shuffling.
This patch will keep the locales in alphabetical order.
MozReview-Commit-ID: GvGYF7j9ftq
--HG--
extra : rebase_source : 6a9aef0efd61c4f1aa7df48ca513311da203ccdb
Stop hardcoding `android`, since we want to use this for desktop too.
We could potentially remove the `android` platform from the bumper configs
at this point.
We strip `-nightly` from the `build_platform` before comparing against the
`l10n-changesets.json` platform list. If we want to support different sets
of ci and nightly locales, we could either:
- point at a second changesets file for ci. This could either be a flatfile
or json; either works. or,
- explicitly name `win32-nightly` etc. in the platform list, and stop removing
the `-nightly` from the `build_platform` before comparing. This means some
locales may have up to 10 different platforms listed. This may get unwieldy,
but would be explicit.
MozReview-Commit-ID: Fvpby92cXdg
--HG--
extra : rebase_source : 503ce9bd455d9845d6598ce2e06c4a355e737053
This also migrates the vcs.py test to mozversioncontrol and adds a new task for
it.
MozReview-Commit-ID: 9jTRkjNupVA
--HG--
extra : rebase_source : 400f27498e00ea45234ad7c951770b098e916b8e
Add a toolchain job description which calls the
repack_rust.py script to package the requested
upstream build of Rust and its standard libraries
for use in gecko builds.
Links are added to these new toolchains for various build
and analysis tasks as appropriate. The base-toolchain
tasks use an explicitly-versioned toolchain since those
can be different from the current release used for most builds.
The corresponding tooltool manifest entries are removed
now that taskcluster artifact versions are available.
This simplifies the update process since new toolchains
can be packaged and used automatically by just updating
the versions in the task descriptions.
A 'linux64-rust' toolchain can be added to other tasks
as a dependency and artifact. It supports linux64-
hosted builds of Rust code targeting linux64 or linux32.
A 'linux64-rust-macos' toolchain targets linux64-hosted
builds of Rust code targeting macOS on x86_64.
A 'linux64-rust-android' toolchain targets linux64-hosted
builds of Rust code targeting various Android architectures.
Two 'win64-rust' and 'win32-rust' toolchain tasks create
similar entries for Windows-hosted builds. All our automation
builds are hosted on win64, so we could use one artifact
with support for both targets, but currently this doesn't
work because of cross-compilation issues in some crates.
This patch maintains the previous separation between
win32 and win64 rust toolchains until that can be addressed.
MozReview-Commit-ID: GRiJml8CtzO
--HG--
extra : rebase_source : 09a3698ce7f9a8b5f2b5d9b5a1fde9c05dc6b540
Copy the repack_rust.py from the rust-build docker container
so it can be used more generally by other taskcluster jobs.
Add --host, --target, and --suffix switches, allowing control
of the packaged toolchain and standard library builds from
the command line.
This drops the previous default behaviour of packaging all
supported platforms and targets.
Add a hard-coded copy of the Rust release signing key to
the script and add it to the running user's gpg config
so we can validate downloaded artifacts from the project
in automation.
Remove the keybase artifact validation since it requires
out-of-project network services and doesn't provide much
advantage in automation.
Calculate the SHA-2 checksum during download and remove
the dependency on shasum/sha256sum command-line tools.
Use more python for filesystem an process interaction
in general.
Create a generic rustc.tar.* package to correctly match
the unversioned unpack dirctory name.
Add support for copying the package to an output directory
if the UPLOAD_DIR environment variable is set. This lets
us hook up the script to taskcluster toolchain jobs without
an external wrapper.
MozReview-Commit-ID: 68LmY3QVU8V
--HG--
extra : rebase_source : f6329056d518ad2cd25faa5c71b22130cbc65c8f
Add an optional 'arguments' key to the yaml description for toolchain
tasks. This is a list of strings to be passed to the script invocation.
This lets us set behaviour, e.g. selecting the version to build or
selecting targets from the task description instead of having to
hard-code those things in the build script itself. Where the same
script otherwise works for multiple configurations, that is easier
to update and simplifies supporting variants.
MozReview-Commit-ID: 30oJYnQaZ7A
--HG--
extra : rebase_source : bdd7bdc5f874d1329ff52d900cd1ac93df23c6dc
Run scripts with a `.py` filename extension through `./mach python`
so the normal enviroment with in-tree python libraries is available.
This is helpful for the Rust upstream release download and repackaging
steps, which are more easily expressed in python than in an sh-based
build script like we use for clang and other tools.
Invocation of `mach python` on Windows-hosted generic workers fails
because of some missing environment pieces. For the purposes of this
bug we can just run the script for Windows targets in a docker-worker
so Python support was left unimplemented for generic workers.
MozReview-Commit-ID: 4XUQ1XrVBc2
--HG--
extra : rebase_source : df81d2da7e70fffb29e96377f16ab22def9e94e0
This is a minor cleanup of some of the source-test configs.
MozReview-Commit-ID: EssE4V4VoXx
--HG--
extra : rebase_source : 6d946aba6e8ff8eaa40965d529d28204f43ec7a4
This allows defining a "job-defaults" key in a yaml file specified by
"jobs-from". Defaults defined here will first be merged into kind specified
defaults (if any), and then tasks defined in the same file will be merged into
the combined defaults.
MozReview-Commit-ID: Jy5m4Pc9umx
--HG--
extra : rebase_source : 5ba11b989a9406d98efe0bffb7aee5b9869fb6c6
New upstream release.
Maintain a linux64-rust-1.20 for use verifying the minimum-
supported rust version once we update it.
We still use rust 1.19 to build sccache since there are
network connection issues when it's built with more recent
rust releases.
MozReview-Commit-ID: 9KjdYAAQZRa
--HG--
extra : rebase_source : 48a9512351723c7cc9a3ac1414097811f2d93720
Per schedule, bump the minimum-supported rust version to 1.21.0
two weeks after its stable release so we can use new code which
depends on it.
MozReview-Commit-ID: Bn8UjvTC7uw