We currently vary the cache name for run-task tasks whenever run-task
changes. This allows us to not worry about backwards or forwards
compatibility of caches in run-task tasks.
This strategy doesn't work for out-of-tree Docker images because
the content of run-task cannot be determined at Taskgraph time:
the content of run-task was determined when that Docker image was
built and there is no way to get that content efficiently during
Taskgraph.
So, for out-of-tree Docker images we now vary the cache name by
the Docker image value, which includes its name and a tag or
hash. This means that out-of-tree run-task tasks will get separate
caches for each distinct Docker image.
This isn't ideal. Ideally we would share caches if run-task doesn't
vary between Docker images. But without any way of proving that
at Taskgraph time, we take the safe road and force cache separation.
MozReview-Commit-ID: FMiQBqfvjqW
--HG--
extra : rebase_source : b2763625a3a69e0cf11b6d648a6fcca379234f02
The image_builder Docker image doesn't set a "command" in its task
definition. The image instead relies on a RUN in its Dockerfile to
control the started command. This command is a shell script which
eventually runs run-task.
This all means that image_builder tasks are executing run-task but
the cache sanitization implemented in bug 1391476 isn't getting
applied to those tasks. This means run-task could barf due to
constraint violations due to improperly configured caches.
The fix for this is to teach the generic task transform that
image_builder tasks use run-task. The effect of this is that
some environment variables get set and the cache name changes
depending on the contents of run-task.
MozReview-Commit-ID: IFqsDxD0eDh
--HG--
extra : rebase_source : 280983eae7d6a44dfd70f0da8ce325e90e9555c4
This creates a new "job-from" field that contains the relative filename the job was defined
in. The filename is relative to 'config.path'. If the task came from the 'jobs' key defined
in kind.yml, this field will be set to 'kind.yml'.
MozReview-Commit-ID: 9e1tEb6XuZT
--HG--
rename : taskcluster/ci/source-test/python-tests.yml => taskcluster/ci/source-test/python.yml
extra : rebase_source : 89010f00a96f88a4eecc8b37028f26b483b820ec
The Sphinx documentation only needs access to a relatively small number
of files in the repo in order to be generated. It is a good candidate
for using sparse profiles.
This commit defines and uses a "sphinx-docs" sparse profile containing
only the files relevant to Sphinx documentation generation.
There are some quirks with the profile:
* All moz.build files are included. This bloats the profile
by >1000 files. Worse, it realizes directories that have no business
being realized. This clutters the checkout and makes it harder to
find things. There is a moz.build reader that knows how to retrive
file data from Mercurial. We could use that. This feels like follow-up
fodder.
* All mach_commands.py files are included. `mach help` says you can do
things that you aren't able to do in the sparse checkout. There isn't
a good way to add all *.py files while excluding mach_commands.py
files. We /could/ do it with regular expressions. But those are slow.
Let's leave it as is for now and come up with a better solution later.
MozReview-Commit-ID: 7yiqGGE1nAh
--HG--
extra : rebase_source : c148040ea3618e8bfdd369b6f48fc60c6d179285
extra : source : b76e2f6204b20de137f2566dff8121ff3abe5760
Clean up and standardize Treeherder symbols for Talos and AWSY tasks:
* Stylo disabled groups include `sd`
* Stylo sequential groups include `ss`
MozReview-Commit-ID: 7cl6e0XvXNO
--HG--
extra : rebase_source : a7b4e7c78af673e204d7ff7479ce5fc87614248c
Convert all jobs that were exercising Stylo enabled to Stylo disabled instead.
Stylo enabled is now handled by the default jobs.
In Perfherder, Stylo enabled jobs will be untagged and take over the existing
Gecko series. Stylo disabled jobs will have a new `stylo-disabled` tag and
create a new series.
MozReview-Commit-ID: BMXBRg3A95j
--HG--
extra : rebase_source : 0f3a6a1f6807dbb90c00d060a3a7f220c2141c64
Make Linux Stylo Talos jobs explicit in the task list, like they are for macOS
and Windows. This way we avoid have different approaches per platform.
(This is still the Stylo enabled form. The next patch coverts to Stylo
disabled for all platforms.)
MozReview-Commit-ID: K4CcWvR9tlJ
--HG--
extra : rebase_source : 3727ddb09114e56493b5ee42df1b45a830a3a6dc
We no longer need to run unit / integration tests on Stylo sequential, but we'd
like to retain it for memory and performance checks.
In addition, we no longer need to explicitly `--enable-stylo` since it is the
default. This also drops the `stylo` tag in Perfherder.
MozReview-Commit-ID: 3OI8Y7Q5PW
--HG--
extra : rebase_source : d80103b3c7991a833dc4603d1df4a6c4b62fd6c9
The signing pool is a limited resource. We already have logic to avoid re-signing
widevine if the .sig files are there, and we're taking advantage of that on linux
and windows. OSX was the lone desktop os still pointing at the unsigned bits for
l10n.
We're pointing at repackage instead of signing because the l10n logic expects a dmg.
MozReview-Commit-ID: FtWa4o5s7aJ
--HG--
extra : rebase_source : f704a97293a231d26a4806543f53fc7e83eb9ca9
Our current sccache build links in openssl's libraries dynamically. The
sonames of the dynamic libraries linked in are specific to the
CentOS/Fedora-ish systems that we build on; attempting to run the
generated sccache binaries on different systems (e.g. Debian-ish) will
result in failure. All of our current automation images are
CentOS-based, but for various reasons, Debian-based images may be used
in the future, and it would be great to have an sccache binary to run on
such systems as well. (It might also be interesting to distribute the
sccache binary we use to local developers as well, but that's a bit
further off.)
Therefore, this patch alters the sccache build on Linux to use static
linking for openssl. We cannot use the system openssl we build on
because the system openssl links to libkrb5, and the distribution we use
for the system images does not provide static libraries of libkrb5.
Building openssl ourself enables us to eliminate the libkrb5 dependency.
An sccache binary from builds with this patch depends on the following
libraries:
froydnj@hawkeye:~$ ldd sccache2/sccache
linux-vdso.so.1 => (0x00007ffe02b39000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff0e7403000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff0e71fb000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff0e6fdd000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff0e6dc6000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff0e69fc000)
/lib64/ld-linux-x86-64.so.2 (0x0000557c8540b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff0e66f2000)
which are standard on any Linux distribution.
This includes adding TASKCLUSTER_VOLUMES to docker image builds directly. The
env variable is not added as part of the task transform because `run-task` is
not in payload.command. In fact, build-image.sh calls run-task after doing
some other housekeeping.
Ideally image builds would be turned into jobs and all of this would occur
automatically, but that turns out to be quite a bit too complex for this
incidental fix -- perhaps best solved in another bug.
MozReview-Commit-ID: FYHvafJras7
--HG--
extra : rebase_source : 4e3b9ae9900727e7932c13ced34b3f8596d755d9
This adds a new morph template for modifying a task's env and the corresponding glue
to specify it from |mach try fuzzy|. It can be used like:
./mach try fuzzy --env FOO=1 --env BAR=baz
This will simply set those environment variables in *all* tasks. We could add the
ability to only specify it for a subset of tasks in the future, but that seems like
a hard problem that probably isn't worth it.
MozReview-Commit-ID: C4sokv886PU
--HG--
extra : rebase_source : 64c9105848e6becd33fae63aeabc3423cfda3bdf
Remove stylo-disabled branch customizations so that it runs everywhere we expect
for tier 1, including beta and release.
MozReview-Commit-ID: DbbiHTvCCyW
This adds inbound as well as release branches. This test is Linux 64 only, so
the added cost and load is hopefully minimized.
MozReview-Commit-ID: AYSmqj0q6jB
--HG--
extra : rebase_source : 50f5766d310db6a7f44f70a45071ca1b99d596c7
This continues to use a file-based reader when run locally.
MozReview-Commit-ID: CJuYKDj2E3n
--HG--
extra : rebase_source : 8532c2cd8bff035b6b7f497947356383713944d7
This means that a push to try affecting only Android will only run android builds
and tests, for example.
MozReview-Commit-ID: HVUvIg0EUZn
--HG--
extra : rebase_source : 80a41a86ba892ccfe56bcbf6b4daf03a60424a5f
extra : source : b41cd667697e13c989659b16bf649090a3908ecd
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : 4cf3efc9c57bb14d2f44147c8881d0a0a18703d6
extra : source : 046d705929f7a41e977eec19c8503afccdec7592
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : 03a10610aa3337269fe76a1196bb9b1665e1ab20
extra : source : b53ff084c2d7968a1d9864d1343f2d9381fb652b
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
--HG--
extra : rebase_source : c6e2ff90316d43cd93826de5c30a1936f19c01ca
extra : source : c25af2c111a5be4e3381d0b002641691d15fe4e8
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
--HG--
extra : rebase_source : d5bce42fc0ea24616d885eed62e5e5a42b4fce24
extra : source : a3b7bdfdb116300daa3f49e0dfc96177e1369440
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
Previously, we uploaded the main Firefox tree docs to /.
In reality, there are multiple Sphinx projects in the repo. In
addition, it is sometimes desirable to access docs for an older
version of Firefox.
In this commit, we add support for specifying the S3 key prefix
for uploads. Then we change the upload code to upload to multiple
locations:
* <project>/latest (always)
* <project>/<version> (if a version is defined in the Sphinx config)
* / (for the main Sphinx docs project)
For the Firefox docs, ``version`` corresponds to a sanitized value from
``milestone.txt``. Currently, it resolves to ``57.0``.
While we're here, we add support for declaring an alternate project
name in the Sphinx conf.py file. If ``moz_project_name`` is defined,
we use that as the project name. For Firefox, we set it to ``main``.
This means our paths (local and uploaded) are now ``main`` instead of
``Mozilla_Source_Tree_Docs``. That's much more pleasant.
MozReview-Commit-ID: 8Gl6l2m6uU4
--HG--
extra : rebase_source : e56885092c12eb8cc76e5e7300f938be566e3e5a
extra : intermediate-source : 8509af1e135177a93460270b27f263c10a62d996
extra : source : 71b4f32caf209fe9dffc340c0b8ccb51ac79c7de
We now have an --upload flag to control whether upload is performed.
We don't inline it because we want to maintain a "firewall" between
regular docs and all the extra packages and imports needed for S3.
MozReview-Commit-ID: DVKhsS545gp
--HG--
extra : rebase_source : 7fee832145189be882db0f0ca057eda6158e0492
By using mozpack, we get deterministic archives. This also makes the
task command simpler.
MozReview-Commit-ID: EPI7tuGQuso
--HG--
extra : rebase_source : babcbf3d5607480321106b40eaa4598c0faee188
We want to verify that Sphinx docs work on all repos. It is only
uploading that should be limited.
MozReview-Commit-ID: An6EZ7jpaWh
--HG--
extra : rebase_source : 86400968630b3ab1c09c43d5fa42d456cec159c4
This fixes the issue where "-u web-platform-tests-wdspec" scheduled
nothing. Now it will schedule a e10s-enabled run of wdspec tests.
MozReview-Commit-ID: AdHZqtk1hLy
--HG--
extra : rebase_source : 8d5926277b37952616c4dffdb20133868913bab2
The updated Docker image contains robustcheckout and run-task support
for sparse checkouts, which are obvious prerequisites.
We change the cache name so sparse and non-sparse checkouts don't
use the same working directory. If we didn't do this, tasks running
from images with old Mercurial clients or without a sparse aware
robustcheckout would fail.
The effect of using a sparse checkout is that we reduce the number
of files in the checkout from ~234,000 to ~3,600. This reduces time
for a fresh checkout from several dozen seconds to under 2s.
MozReview-Commit-ID: IJz794g8ZKH
--HG--
extra : source : 9923fffd4f64a1aa9d762e6027e0e2424a19c49c
The updated Docker image contains robustcheckout and run-task support
for sparse checkouts, which are obvious prerequisites.
We change the cache name so sparse and non-sparse checkouts don't
use the same working directory. If we didn't do this, tasks running
from images with old Mercurial clients or without a sparse aware
robustcheckout would fail.
The effect of using a sparse checkout is that we reduce the number
of files in the checkout from ~234,000 to ~3,600. This reduces time
for a fresh checkout from several dozen seconds to under 2s.
MozReview-Commit-ID: IJz794g8ZKH
--HG--
extra : rebase_source : d262c8314381a136cf5cdc5c33669c8c61818d1d
`run-task` is taught a --sparse-profile argument to be passed down
to `hg robustcheckout` for the main source checkout. It does what
you expect: performs a sparse checkout using the named profile.
The Taskgraph YAML for run-task is taught a "sparse-profile"
property to define the sparse profile. When defined, --sparse-profile
will be passed down to `run-task` and the cache name will be updated
to reflect the use of sparse checkout.
Our cache checking transform is updated to audit for the use of
--sparse-profile without the corresponding "-sparse" cache name
variation.
The reason we need a distinct cache name for sparse is because
clients that aren't sparse aware will be unable to read checkouts
that are sparse. By forcing sparse and non-sparse into different
cache pools, we avoid compatibility issues.
In the ideal world, we probably support sparse profiles on all the
VCS checkouts that `run-task` supports (e.g. --tools-checkout).
Perfect is the enemy of done. All of this is defined in-tree and
it is easy enough to change atomically.
MozReview-Commit-ID: 79k7Vul0hHO
--HG--
extra : rebase_source : babe9b42e2796c2341bffc6ecfe829f4daff9e0f
Include "error:" so Treeherder log parser picks things up (hopefully).
Also, actually include cache path in string.
MozReview-Commit-ID: 8b8ou7TJYZs
--HG--
extra : rebase_source : 52b43dd18419a6a8cfb096a6bf4262852c7c0ccc
* eliminate heading for test kinds, of which there is now only one
* make the caches document have a single heading in the TOC
* break out mach commands into a separate document, add ./mach taskgraph morphed
* remove docs for YAML templates support (the .yml file wasn't actually
used -- I expect it was a merge leftover); these are still used for actions.yml,
but once that is gone the code should be removed, too.
* break try out into its own document, edit to distinguish "how to run try"
from "how to generate config"
MozReview-Commit-ID: 76ZopWA9TPL
--HG--
extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
The UID and GID that a task executes under is dynamic. As a result,
caches need to be aware of the UID and GID that owns files otherwise
subsequent tasks could run into permission denied errors. This is
why `run-task --chown-recursive` exists. By recursively changing
ownership of persisted files, we ensure the current task is able
to read and write all existing files.
When you take a step back, you realize that chowning of cached
files is an expensive workaround. Yes, this results in cache hits.
But the cost is you potentially have to perform hundreds of thousands
of I/O system calls to mass chown. The ideal situation is that
UID/GID is consistent across tasks on any given cache and
potentially expensive permissions setting can be avoided. So, that's
what this commit does.
We add the task's UID and GID to run-task's requirements. When we
first see a cache, we record a UID and GID with it and chown the
empty cache directory to that UID and GID. Subsequent tasks using
this cache *must* use the same UID and GID or else run-task will
fail.
Since run-task now guarantees that all cache consumers use the same
UID and GID, we can avoid a potentially expensive recursive chown.
But there is an exception. In untrusted environments (namely Try),
we recursively chown existing caches if there is a uid/gid mismatch.
We do this because Try is a sandbox and any random task could
experiment with a non-standard uid/gid. That populated cache would
"poison" the cache for the next caller. Or vice-versa. It would be
annoying if caches were randomly poisoned due to Try pushes that
didn't realize there was a UID/GID mismatch. We could outlaw "bad"
UID and GIDs. But that makes the barrier to testing things on Try
harder. So, we go with the flow and recursively chown caches in
this scenario.
This change will shine light on all tasks using inconsistent UID
and GID values on the same cache. Bustage is anticipated.
Unfortunately, we can't easily know what will break. So it will be
one of those things where we will have to fix problems as they arise.
Fortunately, because caches are now tied to the content of run-task,
we only need to back out this change and tasks should revert to caches
without UID and GID pinning requirements and everything will work
again.
MozReview-Commit-ID: 2ka4rOnnXIp
--HG--
extra : rebase_source : ccb2b0a9230694f989775b26d5276fd3ac928af3
extra : source : 083d2e1cc8fe44b04e44f74bda3dd8bc75ba826c
run-task's --chown and --chown-recursive are only used on volumes and
caches - the only locations that aren't controlled by the Docker image
itself and thus whose permissions could be "undefined."
Previous commits have taught run-task about the locations of all caches
and volumes. Therefore, we no longer need to manually define paths to
chown. Instead, we can chown as a side-effect of the path being a
cache or a volume.
So, this commit changes run-task to chown caches and volumes
automatically. Since we no longer have a use for --chown and
--chown-recursive, those arguments are removed.
There /could/ be some paths that are caches or volumes but aren't
getting defined as such in Taskgraph. I consider this a bug in
Taskgraph and the recourse is to properly define a path as a cache or
a volume there.
MozReview-Commit-ID: 1yqrhjil6gy
--HG--
extra : rebase_source : 82c9ccb33f09d3ce50a6b7c6021e1e5b77d1b738
We recently introduced support for telling run-task about caches so
it could sanitize them automatically. We also recently taught
docker-worker and docker-engine how to declare volumes.
Building on that work, we now pass a list of paths corresponding
to Docker volumes to run-task.
run-task now verifies volumes behave as expected. Unless the volume
paths correspond to caches, run-task verifies they are empty and chowns
them to an appropriate owner.
Requiring empty volumes is an arbitrary decision. But as the inline
comment says, it keeps things simpler and makes caches and volumes
behave more like each other.
MozReview-Commit-ID: 5lm2uIitrS3
--HG--
extra : rebase_source : abb0be1a36f24d8d818515885dd2a75d13aed48a
We're about to ban files in Docker volumes so they behave almost
identically to caches (which start empty).
We move the install of nexus.xml from Docker image time to
task time. This also means that changes to nexus.xml don't result
in having to rebuild the Docker image.
MozReview-Commit-ID: JIjeJN4mt2
--HG--
rename : taskcluster/docker/android-gradle-build/nexus.xml => taskcluster/scripts/builder/build-android-dependencies/nexus.xml
extra : rebase_source : 53848f06820bda7979b2ae15456e07f8aed2363d
See the inline comment for the rationale here.
This check may not catch all volumes and caches. But after subsequent
commits refactor how permissions for caches and volumes are handled,
this edge case will likely result in permissions errors in the task,
so it isn't worth worrying about.
Several Dockerfile have been updated to add missing VOLUME so the check
passes.
In the case of desktop1604-test, we stopped removing
/home/worker/.cache because you can't remove a mount point, which is
what volumes are inside Docker containers.
MozReview-Commit-ID: GEyNkkX00kN
--HG--
extra : rebase_source : 19b3165d8c6e4431aa43dca51828be255e82e34f
Docker volumes are host-mounted filesystems. We typically mount
caches at their location. But not always. The reason we define
VOLUME in Dockerfiles is we're guaranteed to get a fast host
filesystem instead of AUFS when a cache isn't mounted.
In this commit, we teach the docker-worker payload builder about
the existence of Docker volumes. Docker volumes can be declared
inline in the YAML. More conveniently, we automatically parse out
VOLUME lines from corresponding in-tree Dockerfile.
We'll do useful things with this data in subsequent commits.
MozReview-Commit-ID: BNxp8EDEYw
--HG--
extra : rebase_source : 1aa6dbb000386cd2789d526708ce369640d532c9
Previously, we conditionally added caches to a task if the current
parameters warranted it.
In order to audit that all caches fulfill basic requirements, we need
to have unconditional knowledge of all caches.
This commit introduces an optional key on each cache entry stating
whether it should be skipped in "untrusted" environments. When we
convert a task definition to a worker payload, we filter out these
caches if necessary.
This change uncovered an inconsistency with filtering caches. In
one location we filtered on the source repo name. In others, we
filtered on the SCM level.
Setting the caches in the spidermonkey kind also changed slightly
to ensure we're not overwriting existing caches. I don't think this
has any behavior changes. But the new method is more correct.
MozReview-Commit-ID: 1crpdWHqQ68
--HG--
extra : rebase_source : 69469014dcc92d4de98d5e887b10542af00e05be
~ in paths is mostly a shell-ism. Let's not use it.
The real reason for this is it interfes with upcoming commits that
audit cache and volume paths.
MozReview-Commit-ID: AhjMwg5gexx
--HG--
extra : rebase_source : dd997a5d9c1123beecaa14bc5458a9338b3e07db
Static analysis in my editor was complaining about the old pattern.
Why not fix it while I'm here.
MozReview-Commit-ID: HtrGenolNXb
--HG--
extra : rebase_source : e3ffc4f7961dfd90767b6d1af2b93ad7551de3be
This needs to be globally enabled to support sparse checkouts
and accessing repos using sparse checkouts.
Having the extension enabled should no-op unless sparse checkouts
are being used. i.e. it should be harmless to globally enable.
MozReview-Commit-ID: AKNUOXfYQPx
--HG--
extra : rebase_source : d32b8a89c56c39923d7b0cd61583c2828a29a872
We use Mercurial 4.3.1 pretty much everywhere in CI now.
Mozharness should be testing with it as well.
MozReview-Commit-ID: HT2rocEvdIe
--HG--
extra : rebase_source : 148a9cd82b18e693ee570f31fc961373e8466a3c
The GCC plugin used for hazard builds was built for GCC 4.9. Until a new
plugin is built by a toolchain job for the "default" version of GCC we
build Firefox with, the builds using the plugin need to stay with the
version the plugin was built for.
This corrects a bug introduced in 265873cf1388.
MozReview-Commit-ID: LkZlTVAM17E
--HG--
extra : rebase_source : bf442e620abd6b47adee3d4c56e6f0c19964aea1
We added this versioning as a mechanism to create clean breaks when
version control mechanisms change in significant ways. Now that
run-task's content influences cache names and this cache name
pattern must be used with run-task, we no longer need this
versioning and can rely on run-task's content and cache requirements
to switch to new caches.
MozReview-Commit-ID: KJ7IF7UzDsg
--HG--
extra : rebase_source : 9d562ec3977fea7920039a66be13cfae8980475e
run-task just grew features to aid with cache validation.
Attempts by run-task to use caches not under its control will fail.
So, we add a transform that audits for and ensures that certain
caches are only being used with run-task. This will help catch
stragglers attempting to use e.g. the legacy VCS checkouts or
tooltool caches without run-task. Fortunately, there are no
violations for this policy. Yay!
MozReview-Commit-ID: LBCmDUdgcuM
--HG--
extra : rebase_source : 9a290470088a8d55693dfbadc5186a018d3669b6
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
I must have been in a closure mood when I wrote this code. The
main function is getting a bit heavyweight. So let's extract
these closures to make things less dense.
MozReview-Commit-ID: 4p5yKB1tTxn
--HG--
extra : rebase_source : 3c3e0b352da6290043013aa36c783e21e01460ba
extra : source : 053f0b0b48635c6a87aefe15ad73f361f0f64f79
So we can use it from another module without feeling dirty.
MozReview-Commit-ID: 1TCk9dda6mL
--HG--
extra : rebase_source : 15c2ee0e4d6aa812ab8acc3564da0c3e7ba45e19
Caches shared across levels scare me, even if readers are purported to
perform content verification. We shouldn't take any risks with released
Firefox builds being contaminated by e.g. Try tasks.
Also, the old cache name interferes with my desire to make cache
names dynamic. This requires dynamic scopes. We already have
have level-{{level}}-* scopes for caches. So having all caches
prefixed with this makes things flexible.
MozReview-Commit-ID: LsrcxIYoEh1
--HG--
extra : rebase_source : dfe97f92a726059200ed79afe215ef2cf1fd7bf1
This pattern is repeated a number of times. Let's consolidate it.
A few places set relengapi-proxy=False. AFAICT this was a no-op because
the default value in the worker schema is False. So, these assignments
were dropped.
MozReview-Commit-ID: EoNPBhaMT2J
--HG--
extra : rebase_source : 72b9c6e4b42369720a19bab42c9ea40c942b5494
bug 1382280 tripled the size of desktop-build image because of
installation of debug symbols. It is only used for valgrind, so let's
move valgrind task to use its own image.
MozReview-Commit-ID: 16St7dDj8tr
--HG--
rename : taskcluster/docker/desktop-build/Dockerfile => taskcluster/docker/valgrind-build/Dockerfile
extra : rebase_source : cc66813cab430d906643fbadf63c661e14784f6f
This makes configuration files for yamllint work a bit better. It's still not perfect, but it's an improvement
on the current situation.
MozReview-Commit-ID: IKxgQm1a7bP
--HG--
extra : rebase_source : 051fafe21337f0557ee39ec71c90e74fd61d3da7
This will allow us keep python related linting files in the same place.
MozReview-Commit-ID: ABtq9dnPo9T
--HG--
rename : tools/lint/flake8_/__init__.py => tools/lint/python/__init__.py
rename : tools/lint/flake8_/__init__.py => tools/lint/python/flake8.py
rename : tools/lint/flake8_/flake8_requirements.txt => tools/lint/python/flake8_requirements.txt
extra : rebase_source : 2568bc0bf8f4adbf8e0be73a54d5da068a8d81b0
This includes adding TASKCLUSTER_VOLUMES to docker image builds directly. The
env variable is not added as part of the task transform because `run-task` is
not in payload.command. In fact, build-image.sh calls run-task after doing
some other housekeeping.
Ideally image builds would be turned into jobs and all of this would occur
automatically, but that turns out to be quite a bit too complex for this
incidental fix -- perhaps best solved in another bug.
MozReview-Commit-ID: FYHvafJras7
--HG--
extra : rebase_source : 4e3b9ae9900727e7932c13ced34b3f8596d755d9
This change adds an upload-generated-sources task kind that runs after nightly
builds, fetches their `target.generated-files.tar.gz` artifact, and uploads
all the contained files to an S3 bucket. For actual nightly and release builds
on SCM level 3 trees, the S3 bucket is configured to be publicly accessible,
so that tools like Socorro will be able to fetch generated source files that
appear in crash reports, and debuggers will be able to fetch generated sources
when they show up while debugging Nightly or Release builds.
There are also level-2 and level-1 S3 buckets configured for builds happening
on trees of other levels such as try. They are not configured as publicly
accessible, but they exist so that these tasks can be tested in try.
MozReview-Commit-ID: Js1HRftbtep
--HG--
extra : rebase_source : b1172c9cc8b8be437d3b94a6bf0ff6b2f7d3508b
extra : source : 73bf88110b3821d62a3d393e85b56896a12f2930
Set these as tier 3 right now, because there are frequent intermittent test failures.
MozReview-Commit-ID: Gs6AXLKaVrF
--HG--
extra : rebase_source : b86c6ffd527dc8af963967c2722a3cee66a6ed3f
This will change all build symbols to 'Ba' and set the USE_ARTIFACTS=1 environment variable.
Mozharness will detect this env to decide whether to perform an artifact build or not.
MozReview-Commit-ID: J8HVZzOt4mX
--HG--
extra : rebase_source : 453028d9be5cb2ad07e9a2a8b769cb6aac9893fe
This provides a mechanism to modify the behaviour of tasks from a try push. The try_task_config.json
looks something like:
{
"tasks": ["build-linux64/opt", "test-linux64/opt-mochitest-e10s-1"],
"templates": {
"artifact": {"enabled": 1}
}
}
This tells taskgraph to apply the 'artifact' template to all tasks. Templates are JSONe based
.yml files that live under taskcluster/taskgraph/templates. Taskgraph will render every template
against every task definition. The templates themselves can then use JSONe condition statements to
filter out which tasks they should or shouldn't apply to.
MozReview-Commit-ID: J8HVZzOt4mX
--HG--
extra : rebase_source : 95a78bc56d3f90ff1b34aabd84ed92aff1e3d954
This picks up a fix we need to update the OS X SDK we build with.
MozReview-Commit-ID: 8dvq4JV1o7q
--HG--
extra : rebase_source : a07f13992f30a29ede29a2167e7f1da8d533fd09
Other related changes:
* Only target.{zip,tar.gz,tar.bz} get the widevine signature (not the various installers).
* Linux{32,64}-nightly are now repackaged. Their mar files are not signed during signed step anymore. It now happens after the repackage.
* As a consequence, funsize routes for linux are now set to repackage-signing (instead of signing)
* Signed upstream artifacts are now defined in a dedicated module (to avoid duplication)
* Platforms defined in beetmover_repackage now allow regex (to reduce duplication too)
* Mozharness configs: Delete unused (and misleading) `src_mozconfig` for windows. This value is actually not used when `run_configure` (in the same dict) is set to False.
MozReview-Commit-ID: COKqevW9Mzn
--HG--
extra : source : ffc2e43aa834e05f0d51d68dfb36317c1b408b08
This is pretty straightforward: we just update some version numbers
and hashes.
The tooltool artifacts were produced and uploaded by me, just like
the last ones. I have some patches to establish a proper toolchain
task to build Mercurial. But it is a bit of a rabbit hole due to the
chicken-and-egg problem of Mercurial needing to be in Docker images.
Preserving the existing install mechanism is the simplest path
forward. Plus we need this patch so we can uplift so earlier releases
get a secure Mercurial in their Docker images.
color and pager are enabled by default in 4.2. So remove configuration
options for them that add no value.
MozReview-Commit-ID: 9pkHX044kV8
--HG--
extra : rebase_source : 4b66f05787bc1b46e1e4db2a47439f3d046becf5
The comment removed by this commit invited the potential for badness.
Mercurial 4.3 drops support for Python 2.6 anyway. So let's remove
any indication we support running Mercurial with Python 2.6.
MozReview-Commit-ID: 40K10s95FLg
--HG--
extra : rebase_source : 52251ff6d1e4877b1cd5dcbf4eb75c875cffa452
AFAICT there are no more in-tree references to this image. That
should mean we can nuke it. So do that.
MozReview-Commit-ID: 9LUGjt46ZCi
--HG--
extra : rebase_source : caa9e8f3e355710542794efb7f6f92c2ef43ef0a
The old process ran "before" and "after" steps as root. The
mozharness script doesn't run as root, which required some small
changes to not run Sonatype Nexus as root. Everything else is a
straight-forward move of the scripts out of the `android-gradle-build`
image and into `taskcluster/scripts`.
MozReview-Commit-ID: CqnNI33OKmb
--HG--
rename : taskcluster/docker/android-gradle-build/bin/after.sh => taskcluster/scripts/builder/build-android-dependencies/after.sh
rename : taskcluster/docker/android-gradle-build/bin/before.sh => taskcluster/scripts/builder/build-android-dependencies/before.sh
rename : taskcluster/docker/android-gradle-build/bin/repackage-jdk-centos.sh => taskcluster/scripts/builder/build-android-dependencies/repackage-jdk-centos.sh
extra : rebase_source : f94e6b9b780f96038c60d3825039a0f94add0404
We really want the Android build image to inherit from desktop-build,
but that isn't possible with the current `docker-image: in-tree:`
support. Therefore, way back in the mists of time, I culted
android-gradle-build from desktop-build. This moves it back (mostly)
in line with desktop-build, which has advanced.
MozReview-Commit-ID: 6GmuxHjhAbv
--HG--
extra : rebase_source : 265937bc9ba3bc4c18756b6c675100a62929bafe
There's little value in a more complicated process for setting the
underlying Docker image for a `run: using: mozharness` job.
This uncovered a typo (extraneous comma) in a Mac OS X repackage job.
MozReview-Commit-ID: DGOs1NvuS32
--HG--
extra : rebase_source : cd7a8402259139f37ec649bb5dc39317fe2fc3cb
Since the buildbot-based Windows builds using releng.manifest are busted
anyways, there is no reason to keep clang entries in there. Which makes
those manifests identical to clang.manifest, so remote the latter.
--HG--
extra : rebase_source : eef7eca4bafc4e348eadc04d6da2bd17ea20deea
valgrind test will try to load debug information for the modules present
in a stack trace. If it fails to do it, we endup with a stack trace with
only memory addresses.
We install debuginfo for all installed packages and look for all libs
in the system common locations, and try to install the corresponding
debug information package.
These are acomplished with debuginfo-install yum utility script.
MozReview-Commit-ID: 76mHOUKKJud
Bug 1338651 was backed out because when building a newer image, there
was a valgrind leak report that couldn't resolve symbols. Further
investigation showed the valgrind package installed had symbols stripped.
We upgrade valgrind version and build it from source with symbols.
We had to build inside the docker image because we need to run
"make install". Using "make dist" to generate a tar ball will also run
"make docs", and it is hard to make it work because of the outdated
texlive package present in CentOS 6.
We also apply a patch [1] to valgrind correctly generate symbols
for unloaded objects.
[1] https://bugs.kde.org/show_bug.cgi?id=79362#c62
MozReview-Commit-ID: 2IhuJY28Ke3
We should have CI Lint YAML files in the tree.
MozReview-Commit-ID: GN5pOJCXvnz
--HG--
extra : rebase_source : 1dd718a44d103f8c4e25736a7373a342d00e0152
We should have CI Lint YAML files in the tree.
MozReview-Commit-ID: L83j6SODA3w
--HG--
extra : rebase_source : afb1a26e4a6819fc326f3bd1e9b856d6104597a8
