Job tasks are tasks that are not tied to a specific build or test. As such, they cannot be scheduled
with the regular -p/-u try syntax options. There exists a -j try syntax option, to schedule them, which
defaults to running "all" of them if not specified.
However, there is a bug here where they will only default to "all" if a try syntax exists in the commit
message. They will not be considered if a developer pushes to try without a try syntax. This happens
because self.jobs is initially initialized with '[]' and we use None to determine when to schedule "all"
later on.
I want to move towards a world without try syntax, so we should start improving the UX of the no try
syntax use case.
Note: When I say "schedule" here, I mean added to the target set. They may still be optimized away.
MozReview-Commit-ID: 4TrC84RGiaL
--HG--
extra : rebase_source : 8fd50113c37745bc10181e1311cc62d75485723a
We use a Maven repository and the (misleadingly named!) uploadArchives
task because this is the best way to make Android Studio download and
recognize the Javadoc and sources. With this, it's automatic; with a
single AAR file, it's a nightmare of point-and-click configuration.
This patch does a bunch of Gradle hacking to make -javadoc and
-sources JARs; there's nothing special or particularly likely to break
here.
This patch also adds Proguard declarations to the :geckoview library
project. That involves moving a good part of the Proguard
configuration into mobile/android/geckoview. (I also expand upon the
existing configuration.) This should be only a re-arrangement, and
the resulting file is included in the original, so nothing should be
changed.
MozReview-Commit-ID: BGNW1v92J0k
--HG--
extra : rebase_source : 94633d27e8ae6bafa3d6823996355c22d2e2e6eb
* Compress docker images with zstd
* Removed need for context.tar from decision task
* Index images by level rather than project
MozReview-Commit-ID: 4RL4QXNWmpd
--HG--
extra : rebase_source : 677d8030a15af3288866a70fc648a10b22c396a3
We add support for Mac OSX tests in Taskcluster by support
taskcluster-worker macosx engine. We also add a new configuration
blob-upload-branch needed by mochitest-other.
MozReview-Commit-ID: 5ZaYYRueRVr
--HG--
extra : rebase_source : f81b3d97d44797a323431a640b1f8e29bf192606
Often we need to setup a test configuration for different platforms, but
not for different platform builds (opt/debug). This leads to cumbersome
configuration duplicates. This patch adds support for platform regular
expression matching. For example, if you want to configure the chunk
size for linux64 platform both for opt and debug, originally you would
do this:
chunks:
by-test-platform:
linux64/opt: 4
linux64/debug: 4
default: 8
With regular expression matching, you only need:
chunks:
by-test-platform:
linux64/.*: 4
default: 8
This patch was originally written by Geoffrey Brown for Windows support.
MozReview-Commit-ID: KbMHV7UkTLe
--HG--
extra : rebase_source : 79a4344c7e3e978bb6b93713c6e6e4114ba5d5b8
The `from_parameters` method was never used, and let do confusion over the role
of these parameters. Now there are only two, and they are always required.
MozReview-Commit-ID: AbPqijXucu5
--HG--
extra : rebase_source : 85affd063a543c549afaaa36ce7ee31ed1f943d5
We add support for Mac OSX tests in Taskcluster by support
taskcluster-worker macosx engine. We also add a new configuration
blob-upload-branch needed by mochitest-other.
MozReview-Commit-ID: CxT3BP3sydG
--HG--
extra : rebase_source : 25e16bf5b5d5a8121fd431788185d318612d7ea5
Often we need to setup a test configuration for different platforms, but
not for different platform builds (opt/debug). This leads to cumbersome
configuration duplicates. This patch adds support for platform regular
expression matching. For example, if you want to configure the chunk
size for linux64 platform both for opt and debug, originally you would
do this:
chunks:
by-test-platform:
linux64/opt: 4
linux64/debug: 4
default: 8
With regular expression matching, you only need:
chunks:
by-test-platform:
linux64/.*: 4
default: 8
This patch was originally written by Geoffrey Brown for Windows support.
MozReview-Commit-ID: HFP52N9Ef0k
--HG--
extra : rebase_source : d2a5129b7459fc7f71f59da76d45526cef028e44
This seems like something that the in-tree taskcluster infrastructure
ought to be able to handle, but in the absence of that functionality,
this is the next best option.
This patch provides the implementation that makes it possible to run the devtools test suite with the linux64-jsdcov build to collect js code coverage.
MozReview-Commit-ID: KFmFhKsDq5s
--HG--
extra : rebase_source : 78894caa6b45a0e43fd1a4c29190788523b10e12
The other changesets for this bug worked on Try but failed on
autoland due to permissions failures in /home/worker/workspace.
Ensure the files have the proper permissions.
The path is a cache with the repo and build type keyed into it.
So there should be no problem with this change causing permissions
issues for other tasks.
--HG--
extra : amend_source : 1fa1c2f7a22afabd4b2e252a268b2f492238966c
Do not speak to github.com as part of CI. Do not open ourselves to a
MitM attack. Be more deterministic.
MozReview-Commit-ID: JRGbraQtqJi
--HG--
extra : rebase_source : b03c36f0c604995f396d49ab3d046637ef864613
As part of this, we eliminate tc-vcs since it is redundant with run-task.
MozReview-Commit-ID: 8HAFSM73BOU
--HG--
extra : rebase_source : 92947b92ce23a7ca4aa885d4dfd073d72410ae16
Update the configuration for the external media tests in taskcluster. Use a
taskcluster specific config for the tests rather than the previous buildbot
config, as these are more concise. Also enable the external media tests on
Windows.
MozReview-Commit-ID: 1E5fkFV5DeV
--HG--
extra : rebase_source : 10ddf9378f14d018544f0d26b48c7d5998011dd0
Note that this does not affect task.extra.chainOfTrust, and thus has no impact
on the generated task JSON.
MozReview-Commit-ID: ERSVZlQUMpK
--HG--
extra : rebase_source : 5cee34de62138535eceb4cec7bcb860d45442938
ff5a4bab0813 (bug 1311791) and 332a08725ed0 (bug 1292071) changed
behavior of the VCS caches. First, the store cache / path was merged
into the checkouts cache. Then the path of the Mercurial shared store
was moved within the cache to always be rooted at the cache root.
Caches are shared across tasks. Tasks can execute on any revision
configured to use a cache. So, when interacting with caches, it is
important to consider how every revision configured to use that
cache will interact with it.
Take this scenario for example.
A worker executes a task where the hg shared store is rooted at
/home/worker/checkouts/src/hg-shared. Then the worker executes a
task where the hg shared store is rooted at
/home/worker/checkouts/hg-store. `hg robustcheckout` will see the
checkout from the first task. But then it sees that the store
it is pointing to is at an unexpected location
(checkouts/src/hg-shared instead of checkouts/hg-store). `hg
robustcheckout` aggressively normalizes state to ensure
consistency. So when it sees this mismatch, it blows away the
checkout and creates one from checkouts/hg-store to replace it.
That's a lot of overhead. And this cycle can repeat itself if
the right combination of revisions run on the worker!
A solution to this problem is to create a clean break from caches
when cache semantics change. In TaskCluster, that means using a
different cache.
This commit introduces a "version" component to the checkouts
cache name. By doing so, we create a clean break from all previous
caches, ensuring all revisions this point forward won't encounter
an hg shared store at an unexpected location. This also paves the
road for easily making additional clean breaks in the future.
MozReview-Commit-ID: JT8yuULKpch
--HG--
extra : rebase_source : 2e5d321e4314ff7543423d3c7837b770b7c8a3a3
332a08725ed0 changed the store path logic in a quick and crude manner.
The code could lead to multiple shared stores existing on a cache
if checkouts were in different parent directories.
This commit refactors the code to explicitly declare a path to the
shared hg store. This restores the behavior of ensuring there is only
a single shared store per cache.
MozReview-Commit-ID: 19Aa1QVrVQ8
--HG--
extra : rebase_source : f74c0ab9cba2bf18e335f24c86dd095335961723
This definitely isn't all of it. But I got the obvious chunks. Still
need to clean up mozharness.
MozReview-Commit-ID: JTZBydP3i2r
--HG--
extra : rebase_source : 782401359036751896ef6c153a81a06cb14031bb
We were seeing issues with the Mercurial working directory not being
pristine. While I can't reproduce this, I have a hunch it is due to
mixing and matching stores and checkouts in TaskCluster. For example,
if a worker supports running concurrent tasks and 2 tasks arrive at
the same time, the caches for the store and checkout may look like:
(store0, checkout0)
(store1, checkout1)
However, the next task may get:
(store1, checkout0)
This may confuse Mercurial.
This commit eliminates the "hg-shared" cache and places the shared
stores as a sibling directory of the checkout.
MozReview-Commit-ID: 8SzyS6wWf9C
--HG--
extra : rebase_source : 6205f3fe944a6ad2cb833a5a7c0ae5ba136eaea6
They were timing out after 2 hours. It appears the task was only a few
minutes away from completion. So an extra 30 minutes should be enough.
MozReview-Commit-ID: HbTbWG42iw5
--HG--
extra : rebase_source : 326de6140bda128d8d5a1ecd5b3057249fd4328e
This patch prevents tests which have the 'run_on_projects' attribute assigned to an empty set from running when the try message contains '-p all' and '-u all' together. It also makes the 'match_test' function a little more readable and updates the 'attributes.rst' document to reflect the changes that were made.
MozReview-Commit-ID: IMk0cmSza8U
--HG--
extra : rebase_source : 06d542cbec16bd8091bc282ad1b831fb4671e72a
It doesn't seem good to tie ourselves to the Gecko tooltool manifest for
building clang-cl; we want to stick with something we can update on
clang-cl's schedule, not Gecko's.
We cannot depend on a fixed location for cl.exe in a taskcluster world.
We therefore need to make our build-clang.py script accomodate relative
path names for cc/cxx and assume those are binaries that should be
looked up on PATH.
We also need to modify the Linux build script so that the virtualenv is
used to look up the new 'which' package.
There are a number of changes here:
- Taskcluster machines don't have MSVC installed, so we have to setup all
the MSVC infrastructure ourselves;
- Gecko contains useful Python packages to use, so we need to setup a
python virtualenv to have access to those. Fortunately, mach handles all
of that for us;
- We need to add build tools to our PATH, as they are not pre-installed;
- We need to define UPLOAD_PATH ourself.
The build-clang-windows.sh is basically an exact copy of the Linux one
at the moment; we'll fix it up for all the Windows-specific taskcluster
bits in a future commit.
This leaves the EXTRA_CHECKOUT_REPOSITORIES support in place, but unused. This
will likely be removed when the last vestiges of tc-vcs are removed.
The mozharness scripts for partner builds remain.
MozReview-Commit-ID: CMu5s3nKhTB
--HG--
extra : rebase_source : 32a5ccc135232562d66fbd7e312bdd60a3d6b1c5
We also include the example APK, since it will be helpful to be able
to regression test the example without downloading the AAR and
manually building the example with that AAR.
MozReview-Commit-ID: CMtA1FRS0Rf
--HG--
extra : rebase_source : 42bab43c69093bd008242ec96e74f53bde150583
In particular, this makes a stronger push for readers to consult the source
files, which honestly have all the good stuff in them anyway.
MozReview-Commit-ID: 9dGWQw59h1L
--HG--
extra : rebase_source : fac2fe9324a6a35825ceece948d1960487a75aa5
If you want to run tests on Linux32 and Linux64, you use trychooser syntax like
"-u all[Ubuntu]" which taskcluster supports, but if you also built ASan, and
want tests on ASan, there's no supported documented way to get them. That sounds
like an edge-case, except that to get tests to run on WinXP or Win8 you have to
explicitly list them, so to get tests on every platform you build you have to
list all of them, and if ASan is one of them you have to somehow know to dig
around in the .yml and to know that linux64-asan is the thing to dig out.
MozReview-Commit-ID: 2REf0cUWmK8
--HG--
extra : rebase_source : cb77ff3e016939c94ac05964a6db809fe10aaa1a
The trychooser website advertises 'Android 4.3' as the alias to choose Android
tests in [] syntax, but without support for that in taskcluster, someone who
wants both opt and debug tests would need to guess, with no support or
documentation, that they should use [Android 4.3,android-4.3-arm7-api-15]
MozReview-Commit-ID: 4rEKcvqAfDH
--HG--
extra : rebase_source : 336d3a787569e14e5ae8608902e6e557ef2ce6c3
This transform appears to always be executed on desktop-build.
build.sh always calls checkout-sources.sh and checkout-sources.sh
always performs a Gecko checkout. It /may/ perform a build/tools
checkout unless an environment variable is set.
Now that desktop-build is ready to use run-task, we switch this
transform to run-task. A gecko and tools checkout is *always*
performed. There is room to make the tools checkout optional.
But it completes nearly instantly. So I'm inclined to do this
as a follow-up enhancement, if needed.
Ideally, the checkout would be on a cache so subsequent tasks on
the worker only need to perform a purge+update as opposed to a
full checkout. However, many tasks make assumptions about the
VCS behavior and modifying the checkout path is a non-trivial
amount of work. This can be deferred to a follow-up bug.
MozReview-Commit-ID: 5Auerl4OL9x
--HG--
extra : rebase_source : 7981b19d043bae311620790809c0c1f3f3696bb5
Currently, it is executed via another shell script (build.sh from
the desktop-build image). build.sh is going away and this will be
the main script called by run-task. It already has a shebang. So
make it executable.
MozReview-Commit-ID: GT4Ix4xdx88
--HG--
extra : rebase_source : e1298db624bedf67f00cec5237a5256624cccf62
Previously, SpiderMonkey tasks were using tc-vcs to clone the tooltool
repository so they could have access to tooltool.py. tooltool.py is
already in the execution environment courtesy of being part of the
source checkout. So use it from there.
One less dependency on a 3rd party service at run-time. One less
point for a MITM attack against our automation.
MozReview-Commit-ID: JmavaBG4vii
--HG--
extra : rebase_source : 5f6c56d1398268ceba867bb47645281465949246
When we switch to run-task in a subsequent commit, we can no longer
rely on cwd being /home/worker. So have build-linux.sh change the cwd
explicitly.
MozReview-Commit-ID: 5sYAjHCFi3s
--HG--
extra : rebase_source : 35e1ceafb6dc880f7f28ee152ae7c78d67ca2f29
This was regressed by bug 1296397 which stopped setting the GECKO_HEAD_REPOSITORY and
the like. It wasn't caught because the task doesn't actually depend on those environment
variables except when using the interactive loaner.
This patch supports vcs checkout regardless of whether the task runs from source or not.
MozReview-Commit-ID: CDxdG8G6JDd
--HG--
extra : rebase_source : 8a3d380c86bbfabe6d76854fd8af8e794e75970e
The lint jobs have been running for many weeks and have been stable and
useful throughout. It is time to make them tier 1 tasks.
MozReview-Commit-ID: 5i5KOPhMNKM
--HG--
extra : rebase_source : 556c11aec6ff152c2c8051d8f4a015c9e6fb22b7
In this case, the tasks must have the same schedulerId as the existing task,
so this is calculated using parameters['level'].
MozReview-Commit-ID: G8EE2kvFstT
--HG--
extra : rebase_source : 214885da9b58520727d5f80b9a31bb8a206f9279
These builds can be run on taskcluster to obtain per-test (JSDebugger) code coverage with the linux64-jsdcov build and overall (GCOV) code coverage with the linux64-ccov build. The linux64-jsdcov build also needed to have leak checking disabled for debug mode.
MozReview-Commit-ID: ASgrU2X7RQV
--HG--
extra : rebase_source : b2098e4d01039edd6cff37f3e6a26c2ed3d3d3ba
Currently, build tasks produce a zip file artifact containing the
content of testing/mozharness. This zip file is downloaded and applied
by any test using mozharness (which is most of them).
Now that some tests have a source checkout, we don't need to download
a mozharness zip because we can just use the source checkout.
This commit teaches test-ubuntu.sh to accept a custom path to
mozharness files. If that path is defined, we skip downloading the
mozharness zip.
The taskgraph has been updated to pass the path of mozharness in the
checkout to test-ubuntu.sh when a source checkout is available.
MozReview-Commit-ID: 2P17Wx1oytI
--HG--
extra : rebase_source : a631a87737b253aece3bbc550ce58168212ae87c
extra : source : 21fdf73bbb17e34cfe00e372695c4f21e4ba3e6a
Same deal as the last commit except for config paths.
MozReview-Commit-ID: 3AXwc3i3Iqu
--HG--
extra : rebase_source : 588020b22853f669d9b9ec1b368d82f4b6f14a97
extra : source : dda84d1fb12b3416c81aab675c58b95ecb9f37cd
Mozharness scripts all live in "mozharness/scripts." So, we don't need
to redundantly define this path. Let's define the script as the relative
path within "mozharness/scripts" instead.
IMO this cosmetic improvement can stand on its own. However, the real
reason for this change is to pave the way for running mozharness from
a source checkout instead of downloading a zip archive of it.
MozReview-Commit-ID: FUnMiLlCcGK
--HG--
extra : rebase_source : 8e11f98dc2f63fa7d7fbd893d68e3c6d475a5759
extra : source : 11ffdc56f5becfeb2ee03423f67749f4baf1f3a7
The only difference in these files was the order that pulseaudio is
started and whether compiz is started. We rename test-ubuntu1604.sh
to test-ubuntu.sh, add some distro release detection, and add
some conditional branches so it works on both Ubuntu 12.04 and 16.04.
MozReview-Commit-ID: CaSfuDxss3d
--HG--
rename : taskcluster/scripts/tester/test-ubuntu1604.sh => taskcluster/scripts/tester/test-ubuntu.sh
extra : rebase_source : 29040c1cf7baedda0aaeff4bd788d5d400c127f1
extra : source : f227ea4d52fed84e3e682de0aa4cb8869539d645
Currently, build tasks produce a zip file artifact containing the
content of testing/mozharness. This zip file is downloaded and applied
by any test using mozharness (which is most of them).
Now that some tests have a source checkout, we don't need to download
a mozharness zip because we can just use the source checkout.
This commit teaches test-ubuntu.sh to accept a custom path to
mozharness files. If that path is defined, we skip downloading the
mozharness zip.
The taskgraph has been updated to pass the path of mozharness in the
checkout to test-ubuntu.sh when a source checkout is available.
MozReview-Commit-ID: 2P17Wx1oytI
--HG--
extra : rebase_source : 5d97079f054f3cd236a1c2a3039c3894207377f9
Same deal as the last commit except for config paths.
MozReview-Commit-ID: 3AXwc3i3Iqu
--HG--
extra : rebase_source : c249f77289e24aceaae73dbca0c994b949215814
Mozharness scripts all live in "mozharness/scripts." So, we don't need
to redundantly define this path. Let's define the script as the relative
path within "mozharness/scripts" instead.
IMO this cosmetic improvement can stand on its own. However, the real
reason for this change is to pave the way for running mozharness from
a source checkout instead of downloading a zip archive of it.
MozReview-Commit-ID: FUnMiLlCcGK
--HG--
extra : rebase_source : 3a0093f1995120264fed89dc4af94eea3554bfe5
The only difference in these files was the order that pulseaudio is
started and whether compiz is started. We rename test-ubuntu1604.sh
to test-ubuntu.sh, add some distro release detection, and add
some conditional branches so it works on both Ubuntu 12.04 and 16.04.
MozReview-Commit-ID: CaSfuDxss3d
--HG--
rename : taskcluster/scripts/tester/test-ubuntu1604.sh => taskcluster/scripts/tester/test-ubuntu.sh
extra : rebase_source : 2153d24fbf8208851a6df8728b8a820166278751
These builds can be run on taskcluster to obtain per-test (JSDebugger) code coverage with the linux64-jsdcov build and overall (GCOV) code coverage with the linux64-ccov build. The linux64-jsdcov build also needed to have leak checking disabled for debug mode.
MozReview-Commit-ID: ASgrU2X7RQV
--HG--
extra : rebase_source : af40a6e582665ffcb575092586731f595a362ae4
Set up tooltool to actually use its cache, and remove the unused tc-vcs cache
scope
MozReview-Commit-ID: H1VLxryQPEM
--HG--
extra : rebase_source : b52de7f5a10e5fb4576872bf3169b2faf87ded32
Secrets are accessed via http://taskcluster/secrets, and the hazard scripts
show no evidence of such calls.
MozReview-Commit-ID: 6xp6D2ybwap
--HG--
extra : rebase_source : 5bfe02773f26768c7e79621a43515d6bf8b921ca
This leaves a lot of room for improvement, but gets the job done for now.
MozReview-Commit-ID: HmeTxZjOfOb
--HG--
extra : rebase_source : b6f55134c25cb670409e906da1acc7c0bd8d0dc0
This has some notes for future work on the task, but will work for now.
MozReview-Commit-ID: 7J4tQeKj3KJ
--HG--
extra : rebase_source : d171b5393fa340a47927e8b89ece93392ff23e56