Note that the to_json method prefers the taskgraph's dependencies information
(edges) to that from the task.dependencies entries. At a few points in
task-graph generation, these values differ, although that is expected (for
example, the full task set contains no edges, but that information is still in
task.dependencies). Unifying that representation leads to some difficulty with
task transforms that reach into the dependency tree (beetmover), so the
different representations are left as-is.
MozReview-Commit-ID: GeW8HNwFA9Z
--HG--
extra : rebase_source : 549773e05e18371a399612d9bceccffc29be8cf2
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
Now that we use the real geckolib and have all dependencies vendored,
the dummy geckolib is no longer required, so we remove it.
Also, the taskgraph code for testing for Servo's presence always
passes and is no longer needed, so we remove it.
Pushed on a CLOSED TREE because ¯\_(ツ)_/¯
MozReview-Commit-ID: ITAqArK4Bks
--HG--
extra : rebase_source : 5eedb3994b679109246b89b0456dd2a59ef3212b
extra : amend_source : b0c97486ae2b72fd21c7968849735e4189e2e86f
I want to get Servo vendored into servo/. The previous plan was to
replace the dummy geckolib with the real deal when the vendoring is
done. Unfortunately, this will require a significant `cargo vendor`
change, which we want to punt on for a bit.
So, this commit moves our dummy geckolib outside of servo/ so we
don't need to `cargo update` or `cargo vendor` when the real servo/
is installed.
The change to toolkit/library/rust/shared/Cargo.toml can be reverted
in the stylo repo to allow it to use the real geckolib.
We also update the taskgraph code for detecting Servo. Previously,
it looked for a file in the possibly-vendored servo/ directory. Once
the vendoring happens, this check will always pass. But without the
real geckolib, the Servo builds will fail. So, we change the check
to look for the real geckolib. This is implemented a bit hackily.
But it will be short-lived until we run `cargo vendor`.
MozReview-Commit-ID: CxGTwy6bK9j
--HG--
rename : servo/ports/geckolib/Cargo.toml => toolkit/library/geckolib/Cargo.toml
rename : servo/ports/geckolib/lib.rs => toolkit/library/geckolib/lib.rs
extra : rebase_source : c0e9c867ae74c4eb124e72dc481fd8dc814e65e7
This requires moving the schema utilities to their own util module.
MozReview-Commit-ID: KR5xSJ9ak5Y
--HG--
extra : rebase_source : 1c1f6bfb6a08deb8c0be4b2b58db02d85aafeb89
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: GPFdi0FD6Vn
--HG--
extra : rebase_source : b992786158851f1099aedfce8669a163228edc51
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: DMwRjuV2vpf
--HG--
extra : rebase_source : 211ecf52694078986caf290c5b0cca35c775da61
We add the following command line options to Taskcluster try syntax:
--spsProfile: enable profile mode.
--rebuild-talos <N>: retrigger talos tests N times.
--setenv <VAR>=<val>: add extra environments variables.
--tag <TAG>: run tests only the tag TAG.
--no-retry: doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: EQlE6q5E8z7
--HG--
extra : rebase_source : 4b7323cd915e8ef9820816015b4b45524811eaf1
Considering docker images contents depend very much on the moment they
were built, it is possible that two images with the same hash in the
taskcluster index (at different levels) have different contents. When
this happens, the build or test results could be significantly
different between e.g. try and mozilla-central, possibly leading to
misleading results at landing time.
So if for some reason multiple levels have images for the same hash, the
one used at the highest level should be prefered, such that try uses the
same as mozilla-central once mozilla-central generates the image for
that hash, even if there is an image previously generated for try.
--HG--
extra : rebase_source : 57f593a530da02f9f576872404915c26af544688
We're adding a dummy servo/ports/geckoservo/ directory, which would make
the filtering logic for Taskcluster tasks think that we want to run
Servo tasks all the time. But we don't have a complete installation of
Servo, so things will inevitably fall over if we did that. To avoid
this situation, change the path that the filter checks for to something
a little more specific and less likely to cause conflicts.
This adds `.cron.yml` and a new mach command to interpret it. While
functionality is limited to nightlies right now, there is room to expand to
more diverse periodic tasks. Let your imagination run wild!
MozReview-Commit-ID: KxQkaUbsjQs
--HG--
extra : rebase_source : ddf0a1eadae5a1169c0ead7bcb7b9ce61b255fbf
This adds a HASH file next to the VERSION file in the image
context folders for prebuilt docker images. And uses the
HASH for referencing the image in the tasks created by
the decision task.
This way docker will validate the image hash when pulling it
in production. Thus, attackers won't be able to inject code
by compromising the remote docker registries we use to store
prebuilt images. Further more, this makes validation of the
Chain-Of-Trust artifacts easier as this eliminates the need
for whitelists and hash validation.
MozReview-Commit-ID: FD3B9MyeU9Q
--HG--
extra : rebase_source : e01cdbd0db06b36ba95dec3da936ee307a23aae7
This adds an optional "platforms" key to the job description. It can be used in conjunction with
"by-platform" like so:
platforms:
- linux
- windows
worker-type:
by-platform:
linux: ...
windows: ...
worker:
by-platform:
linux: ...
windows: ...
MozReview-Commit-ID: JwL1NAR4bnY
--HG--
extra : rebase_source : 637dd777912e256d82092ba3e0edd6937cbb03c6
Stylo automation doesn't work unless Servo is present in the source
directory. This commit introduces a "check_servo" filter that prunes
tasks requiring Servo. Currently, this is implemented as a test against
platforms that are unique to Servo.
The use of relative path checking to find the topsrcdir is a bit
unfortunate. But we use this pattern elsewhere in this code.
MozReview-Commit-ID: IRtd53tudJW
--HG--
extra : rebase_source : 8c4742c13878d762fe7970eedfa5937fdaebe8c4
Previously, all callers outside of tests that passed
"target_tasks_method" to TaskGraphGenerator all used the same pattern
of looking for a key in the parameters and calling a function in
the target_tasks module.
Future commits will refactor how target tasks graph work. To
make the transition easier, we move the logic for obtaining the
target tasks method into TaskGraphGenerator.
MozReview-Commit-ID: 3QU09iGhoXh
--HG--
extra : rebase_source : fbcc31d705c4b0e148aa3709ddcb18ad99953231
* Compress docker images with zstd
* Removed need for context.tar from decision task
* Index images by level rather than project
MozReview-Commit-ID: 4RL4QXNWmpd
--HG--
extra : rebase_source : 677d8030a15af3288866a70fc648a10b22c396a3
Often we need to setup a test configuration for different platforms, but
not for different platform builds (opt/debug). This leads to cumbersome
configuration duplicates. This patch adds support for platform regular
expression matching. For example, if you want to configure the chunk
size for linux64 platform both for opt and debug, originally you would
do this:
chunks:
by-test-platform:
linux64/opt: 4
linux64/debug: 4
default: 8
With regular expression matching, you only need:
chunks:
by-test-platform:
linux64/.*: 4
default: 8
This patch was originally written by Geoffrey Brown for Windows support.
MozReview-Commit-ID: KbMHV7UkTLe
--HG--
extra : rebase_source : 79a4344c7e3e978bb6b93713c6e6e4114ba5d5b8
The `from_parameters` method was never used, and let do confusion over the role
of these parameters. Now there are only two, and they are always required.
MozReview-Commit-ID: AbPqijXucu5
--HG--
extra : rebase_source : 85affd063a543c549afaaa36ce7ee31ed1f943d5
Often we need to setup a test configuration for different platforms, but
not for different platform builds (opt/debug). This leads to cumbersome
configuration duplicates. This patch adds support for platform regular
expression matching. For example, if you want to configure the chunk
size for linux64 platform both for opt and debug, originally you would
do this:
chunks:
by-test-platform:
linux64/opt: 4
linux64/debug: 4
default: 8
With regular expression matching, you only need:
chunks:
by-test-platform:
linux64/.*: 4
default: 8
This patch was originally written by Geoffrey Brown for Windows support.
MozReview-Commit-ID: HFP52N9Ef0k
--HG--
extra : rebase_source : d2a5129b7459fc7f71f59da76d45526cef028e44
If you want to run tests on Linux32 and Linux64, you use trychooser syntax like
"-u all[Ubuntu]" which taskcluster supports, but if you also built ASan, and
want tests on ASan, there's no supported documented way to get them. That sounds
like an edge-case, except that to get tests to run on WinXP or Win8 you have to
explicitly list them, so to get tests on every platform you build you have to
list all of them, and if ASan is one of them you have to somehow know to dig
around in the .yml and to know that linux64-asan is the thing to dig out.
MozReview-Commit-ID: 2REf0cUWmK8
--HG--
extra : rebase_source : cb77ff3e016939c94ac05964a6db809fe10aaa1a
In this case, the tasks must have the same schedulerId as the existing task,
so this is calculated using parameters['level'].
MozReview-Commit-ID: G8EE2kvFstT
--HG--
extra : rebase_source : 214885da9b58520727d5f80b9a31bb8a206f9279
