raptor linux tests are currently run on AWS VM images, these should be run on raw hardware
Differential Revision: https://phabricator.services.mozilla.com/D2994
--HG--
extra : moz-landing-system : lando
When iterating on taskgraph changes, the exact number of chunks that
test-verify runs usually isn't important, so skip it when going fast.
Differential Revision: https://phabricator.services.mozilla.com/D2730
--HG--
extra : rebase_source : 4d46eee982e9868050f1201aba74b020045d9ec1
extra : histedit_source : 744948fa80ae8e3b18212e840843906577fd38ec
The taskgraph code for test-verify currently looks at locally changed files to
determine how many chunks should be run. This code exists so that
`mach try fuzzy` show the same chunks that would be run on a try push.
This changes it, so that local commts are only considered on try and when
called from try-select. This makes generating the taskgraph locally faster,
when not using `mach try`. It also makes test-verfiy not consider too many
files, if the try push happens to contain commits that have landed but havent
been pushed to try yet (i.e. the first push to try after a merge, or beta try
pushes).
Differential Revision: https://phabricator.services.mozilla.com/D2698
--HG--
extra : rebase_source : 68b1ea583730ff3086949aa6c7b6a1046b406d23
extra : histedit_source : 68bbc7ca2062c7f425353e6caf6b8959786dc42d
Currently, `mach try fuzzy` generates a taskgraph that is configured exactly
like the most recent push to mozilla-central. This isn't always desirabe, so
pass some configuration down, to allow the taskgraph to behave differently.
Differential Revision: https://phabricator.services.mozilla.com/D2729
--HG--
extra : rebase_source : 99d6958b33211697227e65df17edc1eb337f63a4
extra : histedit_source : 69b5ff6805bc8409340eb71323a1f6fc637259d7
Enable geckoview-junit, mochitest-cl, mochitest-gpu, crashtests, and jsreftests on
the new packet.net platform. Run only on mozilla-central for now, since we have
limited packet.net capacity in place at this time. Run as tier 3 since this is
preliminary, geckoview is under development, and we are not running on integration
branches.
This will apply to cron tasks, action tasks, and decision tasks. It is a
distinct retrigger implementation because (a) we do not want to follow
dependencies, and (b) it takes a lot of scopes to create a decision task, so we
need to limit access to this action.
MozReview-Commit-ID: 21DVSiagcrO
--HG--
extra : rebase_source : 6f027e349e245e4aa4dbed81145a0a5d75218cb1
extra : histedit_source : eff99aee5a0e7496b0734748b29739480eb0e3fb
This additionally reconsiders the order of all of the actions, spacing them 50
"units" apart and putting the more common actions first.
MozReview-Commit-ID: 98IOYKVMcGU
--HG--
extra : rebase_source : 1273a8b86625bd8e4dc3bddab80c6912241f88c8
extra : histedit_source : 16314284a2b4e0368da843b036e22aaedf485307
TOOLTOOL_CHECKOUT is typically `.`, which doesn't work so great for
adding things to $PATH. We need to turn everything we're adding to
$PATH into absolute paths, so $PATH actually works properly.
Otherwise it can't be used as a context manager since it
doesn't have __enter__ or __exit__.
Differential Revision: https://phabricator.services.mozilla.com/D2672
--HG--
extra : moz-landing-system : lando
This commit points the `install_mercurial.sh` script at the
newest Mercurial distributions uploaded to tooltool.
Differential Revision: https://phabricator.services.mozilla.com/D2626
--HG--
extra : amend_source : a9cd500cc86106750aa5b5472e624a00128cac68
extra : histedit_source : 7a06d9c77b6f7a7b2110dc6bc15858c7aa3b7242
This is what a lot of programs do.
We do logging in a helper function so we can flush after every write.
Differential Revision: https://phabricator.services.mozilla.com/D2526
--HG--
extra : rebase_source : 98563aee129c16662a783122241623b8ed2fe457
Previously, we told `tar` or `unzip` to operate on an explicit file.
This worked when `tar` understood the compression format of the file.
And this worked in the majority of cases.
But `tar` does not support zstandard compression (at least not outside
extremely new versions, which aren't yet widely deployed). And not all
versions of `tar` support the `-a` argument.
This commit changes our invocation of `tar` so input data is piped
to it from Python. In the case of `tar`, we perform decompression in
Python, if possible. This allows us to support zstandard and `tar`
binaries that don't support `-a` to auto-detect the compression format.
I wanted to be consistent and always pipe the raw data via stdin.
But `unzip` doesn't appear to like this. Oh well.
We also refactor the logic around detecting archives. We have a
function to identify the archive type based on a filename. We then
pass the archive type to the extraction function and key off that
logic within. We also conditionally call extract_archive() and
fail hard in extract_archive() when things fail. This will make
future archive code easier to reason about.
Differential Revision: https://phabricator.services.mozilla.com/D1576
--HG--
extra : rebase_source : 1c66396cced1b2a94a959386eecc3f512b033308
It was kept on clang 5 explicitly in bug 1467658 because of bug
1467673, now fixed.
--HG--
extra : rebase_source : 8de52e6967bb1f249b7e59d83b90ecfb291a9c44
While fiddling with clang (upgrading it and applying some miscompilation
patches), my mac LTO builds started to fail because ld64 would crash
during configure.
It turns out, it was crashing trying to print a warning it shouldn't
even print out, about failure to create a cache path.
This, in turn, is due to a pointer not being initialized in the ld64
code. I sent this upstream, and this was promptly fixed:
https://github.com/tpoechtrager/cctools-port/pull/57
However, since our last update of cctools-port, upstream landed a change
that broke support for tbd files if you don't compile against the new
libtapi library. Doing so is more work than I'm ready to put here,
so we just cherry-pick the fix.
--HG--
extra : rebase_source : 131952a5233bc379943c8eb124d377525f54202f
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
This also enables the py2 linter which will help maintain compatibility
with both 2 and 3.
Differential Revision: https://phabricator.services.mozilla.com/D1884
--HG--
extra : moz-landing-system : lando
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
These typically run for 25-50 minutes. So, if they are taking longer than that,
there is likely a problem.
Differential Revision: https://phabricator.services.mozilla.com/D2252
--HG--
extra : moz-landing-system : lando
Currently 'fetch' artifacts are all extracted in the same directory, this could
make the extdir messy, or in the worst case, cause file name collisions.
Some artifacts are ok to extract into the same directory as they're already
bundled within the archive. But other artifacts are not. This patch keeps the
default behaviour (extracting everything into the same directory), but allows
task authors to specify per-artifact directories to extract into.
The syntax is:
path[>dest]@<task>
The 'dest' value will be a subdirectory of the MOZ_FETCHES_DIR environment
variable.
Depends on D2102.
Differential Revision: https://phabricator.services.mozilla.com/D2166
--HG--
extra : moz-landing-system : lando
This refactoring will make it easier to set 'run-on-projects' keys across all talos tasks
in one go (rather than needing to modify each task individually).
The job-defaults are recursively (and naively) merged into the task definitions, so each
task that changes 'run-on-projects' also needs to use the 'by-test-platform' key.
Differential Revision: https://phabricator.services.mozilla.com/D2136
--HG--
extra : moz-landing-system : lando
Fetches no longer need to be artifacts exposed via a 'fetch' task, they can
also be artifacts from a task's dependencies. The new format is:
fetches:
fetch:
- fetch-artifact-1.zip
- fetch-artifact-2.zip
build:
- build-artifact-1.zip
...
Specifying 'build' artifacts to fetch will error out if the task doesn't have
any build dependencies.
The 'fetch' key works the same as before, but it is now a special case. Unlike
'build' (or other dependencies), adding a fetch task's artifact here will
implicitly make our task depend on the corresponding fetch task. It will not
be an error.
Depends on D2028.
Differential Revision: https://phabricator.services.mozilla.com/D2102
--HG--
extra : moz-landing-system : lando
Fetches no longer need to be artifacts exposed via a 'fetch' task, they can
also be artifacts from a task's dependencies. The new format is:
fetches:
fetch:
- fetch-artifact-1.zip
- fetch-artifact-2.zip
build:
- build-artifact-1.zip
...
Specifying 'build' artifacts to fetch will error out if the task doesn't have
any build dependencies.
The 'fetch' key works the same as before, but it is now a special case. Unlike
'build' (or other dependencies), adding a fetch task's artifact here will
implicitly make our task depend on the corresponding fetch task. It will not
be an error.
Depends on D2028.
Differential Revision: https://phabricator.services.mozilla.com/D2102
--HG--
extra : moz-landing-system : lando
The 'use_fetches' transform is currently only being used by toolchain tasks,
but we'd like to expand this to more kinds (like 'test' and 'source_test').
The problem is that 'use_fetches' doesn't have a schema, and assumes things
about the kinds of keys that will be set in the job. For example, it assumes
that job['worker']['env'] is going to be forwarded up to the jobdesc properly.
By moving this transform into the set applied to all 'job' tasks, we:
A) Have a task schema we can reliably depend on
B) Can automatically use it from any 'job' task without kind specific
modifications
Since the toolchain tasks apply the 'job' transforms (almost) right after
the 'use_fetches' transform, this change just works.
Differential Revision: https://phabricator.services.mozilla.com/D2028
--HG--
extra : moz-landing-system : lando
This patch makes the QR test platforms tier-1 by default, and removes
the ad-hoc bits that were making individual QR jobs tier-1 before.
However, it also explicitly downgrades some QR jobs to tier-2 or tier-3;
comments in the yml files indicate why.
MozReview-Commit-ID: 1UfPuhcMvIW
--HG--
extra : rebase_source : a2347f6a5929246aaba7656b59c0b8f7aa4ca081
This is needed to not have a circular kind dependency when we actually spell out all dependencies (in a following patch)
Differential Revision: https://phabricator.services.mozilla.com/D1695
--HG--
rename : taskcluster/ci/repackage-signing/kind.yml => taskcluster/ci/repackage-signing-l10n/kind.yml
extra : rebase_source : c2998ba23f213090d27495eb44c3bde3a1628dff
Test verification backfill specifies --gpu-required for certain types
of tests (depending on test path and/or suite). web-platform-tests do
not recognize --gpu-required. This patch updates the backfill logic
to avoid --gpu-required for wpt.
The actionPerm is for access control, so it must limit access to a specific
callback function, not a name (which can apply to mulitple functions).
To make things nicer, we allow functions to specify their cb_name and default
it to the action name. The decorated function names are not used.
MozReview-Commit-ID: 2oiuXrrw7DE
--HG--
extra : rebase_source : 07b27db25e9c8e3226dc996d3fcef401ca498739
Everything but release-promotion (to be handled in another bug) is generic.
For the moment, these all run with the default repo scopes; once this is
landed, I can start adjusting that and granting the necessary scopes only to
these actions.
MozReview-Commit-ID: IB8OEsfeBpj
--HG--
extra : rebase_source : 6ef1697cf255b579097ef8b85be8f9f62718f548
This carefully maintains tasks as an array by putting the conditional inside of
that array. Note that `[{$if: 'false', then: 1}]` returns `[]` in JSON-e --
the missing `else` branch is treated as a missing array element.
MozReview-Commit-ID: 9ARIxW3gfWo
--HG--
extra : rebase_source : 304ce14ccc9abc9f4f48f3179adb981b5fe55a0e
This changes
File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/cancel_all.py", line 30, in list_group
for task in [t['status'] for t in response['tasks']]:
KeyError: u'tasks'
Into a more understandable error (404, in this case).
MozReview-Commit-ID: 5XnFyxIdRfo
--HG--
extra : rebase_source : 797a3117d3246c962f30980c1658fde3bd366135
This will start to cause an error when a newer shellcheck is available in CI
Differential Revision: https://phabricator.services.mozilla.com/D1940
--HG--
extra : moz-landing-system : lando
The resulting action task isn't useful to the user, so instead we send an email
containing a link to the interaction console.
MozReview-Commit-ID: 5uHnQo9WTF6
--HG--
extra : rebase_source : 1213afa7c53a0bcc4a07c4c2970c7bf21ab3b7f1
This also updates actions to "see through" the conditional. Soon we won't be
using kind=task, so this hack will be less important.
MozReview-Commit-ID: Aa6g9ZqoPMa
--HG--
extra : rebase_source : 7434f2047c48ff0d1fa6de9e3419fb4e0bf0bb72
The resulting action task isn't useful to the user, so instead we send an email
containing a link to the interaction console.
MozReview-Commit-ID: 5uHnQo9WTF6
--HG--
extra : rebase_source : ec52a333582a2778c2cec12d612d681e1a9b1976
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : a17aa496bf3d4af4d1349d69a637c686c6817d0f
Also include webgl2-deqp, which we would like to run eventually, but not yet.
MozReview-Commit-ID: CY4hYCI95ws
--HG--
extra : rebase_source : 9973df0f905bb65d2e8b8c66a6a57e8869e527c1
Also include webgl2-deqp, which we would like to run eventually, but not yet.
MozReview-Commit-ID: FDWdu1J0end
--HG--
extra : rebase_source : a47d88cb2c5eb82e4dfaa9e58d76acbf0736d35d
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
Assorted fixes from trawling the sphinx logs - malformed formatting, broken references, leftovers from renaming action-task to action-callback and removing
yaml-templates, docstring fixes to make sphinx happier, and typos.
MozReview-Commit-ID: 6jUOljdLoE2
--HG--
extra : rebase_source : f2a9dbcde5180f760a80f47691a70eba76e58bad
GCC will pick up whatever `as` is first in PATH when trying to assemble
files. It expects this `as` to have at least as many features as the
`as` detected at configure time when GCC was originally built. We
should ensure that GCC is always picking up an appropriate `as` by
adding its base directory to the search path; otherwise, we get peculiar
assembler errors.
Various fixes for the docs to make docs better and sphinx happier
* remove unused targets which were duplicated in balrog, partials
* fix up malformed targets and links
* convert docstrings to comments so sphinx ignores example response in utils/partials.py
* section underlines should match titles
MozReview-Commit-ID: GSYqsocBC4I
--HG--
extra : source : e464e9bbb42fb85170f1ce35a01f25811d753871
