This is what a lot of programs do.
We do logging in a helper function so we can flush after every write.
Differential Revision: https://phabricator.services.mozilla.com/D2526
--HG--
extra : rebase_source : 98563aee129c16662a783122241623b8ed2fe457
Previously, we told `tar` or `unzip` to operate on an explicit file.
This worked when `tar` understood the compression format of the file.
And this worked in the majority of cases.
But `tar` does not support zstandard compression (at least not outside
extremely new versions, which aren't yet widely deployed). And not all
versions of `tar` support the `-a` argument.
This commit changes our invocation of `tar` so input data is piped
to it from Python. In the case of `tar`, we perform decompression in
Python, if possible. This allows us to support zstandard and `tar`
binaries that don't support `-a` to auto-detect the compression format.
I wanted to be consistent and always pipe the raw data via stdin.
But `unzip` doesn't appear to like this. Oh well.
We also refactor the logic around detecting archives. We have a
function to identify the archive type based on a filename. We then
pass the archive type to the extraction function and key off that
logic within. We also conditionally call extract_archive() and
fail hard in extract_archive() when things fail. This will make
future archive code easier to reason about.
Differential Revision: https://phabricator.services.mozilla.com/D1576
--HG--
extra : rebase_source : 1c66396cced1b2a94a959386eecc3f512b033308
It was kept on clang 5 explicitly in bug 1467658 because of bug
1467673, now fixed.
--HG--
extra : rebase_source : 8de52e6967bb1f249b7e59d83b90ecfb291a9c44
While fiddling with clang (upgrading it and applying some miscompilation
patches), my mac LTO builds started to fail because ld64 would crash
during configure.
It turns out, it was crashing trying to print a warning it shouldn't
even print out, about failure to create a cache path.
This, in turn, is due to a pointer not being initialized in the ld64
code. I sent this upstream, and this was promptly fixed:
https://github.com/tpoechtrager/cctools-port/pull/57
However, since our last update of cctools-port, upstream landed a change
that broke support for tbd files if you don't compile against the new
libtapi library. Doing so is more work than I'm ready to put here,
so we just cherry-pick the fix.
--HG--
extra : rebase_source : 131952a5233bc379943c8eb124d377525f54202f
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
This also enables the py2 linter which will help maintain compatibility
with both 2 and 3.
Differential Revision: https://phabricator.services.mozilla.com/D1884
--HG--
extra : moz-landing-system : lando
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
These typically run for 25-50 minutes. So, if they are taking longer than that,
there is likely a problem.
Differential Revision: https://phabricator.services.mozilla.com/D2252
--HG--
extra : moz-landing-system : lando
Currently 'fetch' artifacts are all extracted in the same directory, this could
make the extdir messy, or in the worst case, cause file name collisions.
Some artifacts are ok to extract into the same directory as they're already
bundled within the archive. But other artifacts are not. This patch keeps the
default behaviour (extracting everything into the same directory), but allows
task authors to specify per-artifact directories to extract into.
The syntax is:
path[>dest]@<task>
The 'dest' value will be a subdirectory of the MOZ_FETCHES_DIR environment
variable.
Depends on D2102.
Differential Revision: https://phabricator.services.mozilla.com/D2166
--HG--
extra : moz-landing-system : lando
This refactoring will make it easier to set 'run-on-projects' keys across all talos tasks
in one go (rather than needing to modify each task individually).
The job-defaults are recursively (and naively) merged into the task definitions, so each
task that changes 'run-on-projects' also needs to use the 'by-test-platform' key.
Differential Revision: https://phabricator.services.mozilla.com/D2136
--HG--
extra : moz-landing-system : lando
Fetches no longer need to be artifacts exposed via a 'fetch' task, they can
also be artifacts from a task's dependencies. The new format is:
fetches:
fetch:
- fetch-artifact-1.zip
- fetch-artifact-2.zip
build:
- build-artifact-1.zip
...
Specifying 'build' artifacts to fetch will error out if the task doesn't have
any build dependencies.
The 'fetch' key works the same as before, but it is now a special case. Unlike
'build' (or other dependencies), adding a fetch task's artifact here will
implicitly make our task depend on the corresponding fetch task. It will not
be an error.
Depends on D2028.
Differential Revision: https://phabricator.services.mozilla.com/D2102
--HG--
extra : moz-landing-system : lando
Fetches no longer need to be artifacts exposed via a 'fetch' task, they can
also be artifacts from a task's dependencies. The new format is:
fetches:
fetch:
- fetch-artifact-1.zip
- fetch-artifact-2.zip
build:
- build-artifact-1.zip
...
Specifying 'build' artifacts to fetch will error out if the task doesn't have
any build dependencies.
The 'fetch' key works the same as before, but it is now a special case. Unlike
'build' (or other dependencies), adding a fetch task's artifact here will
implicitly make our task depend on the corresponding fetch task. It will not
be an error.
Depends on D2028.
Differential Revision: https://phabricator.services.mozilla.com/D2102
--HG--
extra : moz-landing-system : lando
The 'use_fetches' transform is currently only being used by toolchain tasks,
but we'd like to expand this to more kinds (like 'test' and 'source_test').
The problem is that 'use_fetches' doesn't have a schema, and assumes things
about the kinds of keys that will be set in the job. For example, it assumes
that job['worker']['env'] is going to be forwarded up to the jobdesc properly.
By moving this transform into the set applied to all 'job' tasks, we:
A) Have a task schema we can reliably depend on
B) Can automatically use it from any 'job' task without kind specific
modifications
Since the toolchain tasks apply the 'job' transforms (almost) right after
the 'use_fetches' transform, this change just works.
Differential Revision: https://phabricator.services.mozilla.com/D2028
--HG--
extra : moz-landing-system : lando
This patch makes the QR test platforms tier-1 by default, and removes
the ad-hoc bits that were making individual QR jobs tier-1 before.
However, it also explicitly downgrades some QR jobs to tier-2 or tier-3;
comments in the yml files indicate why.
MozReview-Commit-ID: 1UfPuhcMvIW
--HG--
extra : rebase_source : a2347f6a5929246aaba7656b59c0b8f7aa4ca081
This is needed to not have a circular kind dependency when we actually spell out all dependencies (in a following patch)
Differential Revision: https://phabricator.services.mozilla.com/D1695
--HG--
rename : taskcluster/ci/repackage-signing/kind.yml => taskcluster/ci/repackage-signing-l10n/kind.yml
extra : rebase_source : c2998ba23f213090d27495eb44c3bde3a1628dff
Test verification backfill specifies --gpu-required for certain types
of tests (depending on test path and/or suite). web-platform-tests do
not recognize --gpu-required. This patch updates the backfill logic
to avoid --gpu-required for wpt.
The actionPerm is for access control, so it must limit access to a specific
callback function, not a name (which can apply to mulitple functions).
To make things nicer, we allow functions to specify their cb_name and default
it to the action name. The decorated function names are not used.
MozReview-Commit-ID: 2oiuXrrw7DE
--HG--
extra : rebase_source : 07b27db25e9c8e3226dc996d3fcef401ca498739
Everything but release-promotion (to be handled in another bug) is generic.
For the moment, these all run with the default repo scopes; once this is
landed, I can start adjusting that and granting the necessary scopes only to
these actions.
MozReview-Commit-ID: IB8OEsfeBpj
--HG--
extra : rebase_source : 6ef1697cf255b579097ef8b85be8f9f62718f548
This carefully maintains tasks as an array by putting the conditional inside of
that array. Note that `[{$if: 'false', then: 1}]` returns `[]` in JSON-e --
the missing `else` branch is treated as a missing array element.
MozReview-Commit-ID: 9ARIxW3gfWo
--HG--
extra : rebase_source : 304ce14ccc9abc9f4f48f3179adb981b5fe55a0e
This changes
File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/cancel_all.py", line 30, in list_group
for task in [t['status'] for t in response['tasks']]:
KeyError: u'tasks'
Into a more understandable error (404, in this case).
MozReview-Commit-ID: 5XnFyxIdRfo
--HG--
extra : rebase_source : 797a3117d3246c962f30980c1658fde3bd366135
This will start to cause an error when a newer shellcheck is available in CI
Differential Revision: https://phabricator.services.mozilla.com/D1940
--HG--
extra : moz-landing-system : lando
The resulting action task isn't useful to the user, so instead we send an email
containing a link to the interaction console.
MozReview-Commit-ID: 5uHnQo9WTF6
--HG--
extra : rebase_source : 1213afa7c53a0bcc4a07c4c2970c7bf21ab3b7f1
This also updates actions to "see through" the conditional. Soon we won't be
using kind=task, so this hack will be less important.
MozReview-Commit-ID: Aa6g9ZqoPMa
--HG--
extra : rebase_source : 7434f2047c48ff0d1fa6de9e3419fb4e0bf0bb72
The resulting action task isn't useful to the user, so instead we send an email
containing a link to the interaction console.
MozReview-Commit-ID: 5uHnQo9WTF6
--HG--
extra : rebase_source : ec52a333582a2778c2cec12d612d681e1a9b1976
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : a17aa496bf3d4af4d1349d69a637c686c6817d0f
Also include webgl2-deqp, which we would like to run eventually, but not yet.
MozReview-Commit-ID: CY4hYCI95ws
--HG--
extra : rebase_source : 9973df0f905bb65d2e8b8c66a6a57e8869e527c1
Also include webgl2-deqp, which we would like to run eventually, but not yet.
MozReview-Commit-ID: FDWdu1J0end
--HG--
extra : rebase_source : a47d88cb2c5eb82e4dfaa9e58d76acbf0736d35d
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
Assorted fixes from trawling the sphinx logs - malformed formatting, broken references, leftovers from renaming action-task to action-callback and removing
yaml-templates, docstring fixes to make sphinx happier, and typos.
MozReview-Commit-ID: 6jUOljdLoE2
--HG--
extra : rebase_source : f2a9dbcde5180f760a80f47691a70eba76e58bad
GCC will pick up whatever `as` is first in PATH when trying to assemble
files. It expects this `as` to have at least as many features as the
`as` detected at configure time when GCC was originally built. We
should ensure that GCC is always picking up an appropriate `as` by
adding its base directory to the search path; otherwise, we get peculiar
assembler errors.
Various fixes for the docs to make docs better and sphinx happier
* remove unused targets which were duplicated in balrog, partials
* fix up malformed targets and links
* convert docstrings to comments so sphinx ignores example response in utils/partials.py
* section underlines should match titles
MozReview-Commit-ID: GSYqsocBC4I
--HG--
extra : source : e464e9bbb42fb85170f1ce35a01f25811d753871
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : c788ef4f7da9949b81df2f0577af6f6039ea63d8
We shouldn't run this on central, as it falls back to the dev configs, and fails.
It should be fine on beta/release/esr60. I had to move this version of the check to its own
kind to avoid the dependency tree bringing in the entire build process. Perhaps we can
refactor later to avoid duplication
Differential Revision: https://phabricator.services.mozilla.com/D1765
--HG--
rename : taskcluster/ci/release-bouncer-check/kind.yml => taskcluster/ci/bouncer-check/kind.yml
yaml.load() can evaluate arbitrary Python code via syntax such as
`!!python/object/apply:os.system`. Seriously.
Let's switch taskgraph to yaml.safe_load(), which is reasonable
about limiting magic.
Differential Revision: https://phabricator.services.mozilla.com/D1736
The max-run-time for tasks appears to have been mostly cargo
culted. In particular, a value of 36000 (10 hours) is quite absurd:
no single task takes that long to execute.
This commit reduces the max-run-time of several tasks to
more reasonable values. The goal here is to prevent
excessively long-running tasks. There is definitely room to
further tweak values to further mitigate long-running tasks. But
let's bite off the biggest chunk first, since that doesn't
require much mental effort.
There is a possibility I overshot on some of these tasks. If we
get timeouts, we can always increase the timeout again.
Differential Revision: https://phabricator.services.mozilla.com/D1716
