Some debug info for system packages are not available to valgrind in the
valgrind docker image, some of which are for libraries actively used by
Firefox. Not all of them, unfortunately, have debug info available, but
some of them do and we add them all. We could skip a few that are not
really useful, but that doesn't make a significant difference to the
docker image size (~0.3%).
Differential Revision: https://phabricator.services.mozilla.com/D32419
--HG--
extra : moz-landing-system : lando
Generating partials from betas doesn't work if we check the 'from' mar channel IDs, as we don't know what's valid for those.
Differential Revision: https://phabricator.services.mozilla.com/D27866
--HG--
extra : moz-landing-system : lando
This also switches it to use the generic toolchain build image, as
it is no longer being used exclusively by mingw builds.
Differential Revision: https://phabricator.services.mozilla.com/D24230
--HG--
extra : moz-landing-system : lando
Run checks done in push-apk in promote-phase, instead of the very last task of the pipeline
Differential Revision: https://phabricator.services.mozilla.com/D26328
--HG--
rename : taskcluster/docker/google-play-strings/Dockerfile => taskcluster/docker/mozapkpublisher/Dockerfile
extra : moz-landing-system : lando
When docker images use setup_packages.sh, they add apt sources. While we
currently do run apt-get update to pick those new sources, if a package
provided by them is already installed and not explicitly listed in
subsequent apt-get install, they're not going to be upgraded.
Differential Revision: https://phabricator.services.mozilla.com/D26100
--HG--
extra : moz-landing-system : lando
This also switches it to use the generic toolchain build image, as
it is no longer being used exclusively by mingw builds.
Differential Revision: https://phabricator.services.mozilla.com/D24230
--HG--
extra : moz-landing-system : lando
This allows the confined snap to interact with Universal 2nd Factor devices, such as Yubikeys.
Differential Revision: https://phabricator.services.mozilla.com/D24147
--HG--
extra : moz-landing-system : lando
comm-task-env runs before run-task and updates the environment with GECKO_*
variables that are defined in a file at the root of a subproject's repository,
such as "comm-central".
Updates:
- add comm-task-env
- add python 3.5 (run-task dependency)
- add pyyaml
Differential Revision: https://phabricator.services.mozilla.com/D16934
--HG--
extra : moz-landing-system : lando
This will allow to use bash constructs in pre-diff-commands, like
`{a,b}`.
Depends on D23075
Differential Revision: https://phabricator.services.mozilla.com/D23076
--HG--
extra : moz-landing-system : lando
Reinsert awscli for partials, which is needed for caching. Also update packages and fix the metrics recording
Differential Revision: https://phabricator.services.mozilla.com/D22942
--HG--
extra : moz-landing-system : lando
Reinsert awscli for partials, which is needed for caching. Also update packages and fix the metrics recording
Differential Revision: https://phabricator.services.mozilla.com/D22942
--HG--
extra : moz-landing-system : lando
As of the update snapshot, stretch-backports contains version 112.
Depends on D22264
Differential Revision: https://phabricator.services.mozilla.com/D22265
--HG--
extra : moz-landing-system : lando
When the apt snapshot is more recent than the docker image on the docker
hub, some packages may not be up-to-date.
Depends on D22455
Differential Revision: https://phabricator.services.mozilla.com/D22263
--HG--
extra : moz-landing-system : lando
Because the debian9-base apt configuration doesn't install recommended
packages, we end up needing to install more packages than before. We
could pass --install-recommended to apt-get, but that would make the
image larger than it already was after the upcoming changes, because
new versions of diffoscope come with more recommended dependencies.
The side effect is that this makes the image much smaller than it used
to be, while preserving all the useful recommended packages (we don't
actually need all of them).
Differential Revision: https://phabricator.services.mozilla.com/D22262
--HG--
extra : moz-landing-system : lando
I've set HGPLAIN in the container entrypoint so that people running
the main script interactively still get the nice features.
Differential Revision: https://phabricator.services.mozilla.com/D21343
--HG--
extra : moz-landing-system : lando
The build script currently is doing some unnecessary steps:
- Running `gclient` only prints out an help message. It is a step
indicated in various documentations, but is only necessary to keep
depot-tools up-to-date, which they are, since we just cloned it.
- The `fetch v8` command creates a v8 directory, no need to create
another layer.
- `gclient sync` is run as part of `fetch`. Same as `gclient`, this step
is only given in documentations to keep things up-to-date on an existing
clone, but we just freshly got one.
- Same goes for `git pull && gclient sync`
- `git checkout master` is not necessary, as `fetch` gets us there
already (albeit, in a detached head state)
- install-build-deps.sh installs build dependencies for chrome or
whatever. That's way too much for v8, that barely needs pkg-config and
glib, which we now install in the docker image.
Differential Revision: https://phabricator.services.mozilla.com/D20082
We however leave moving the packages building to a script for another
day.
Differential Revision: https://phabricator.services.mozilla.com/D19624
--HG--
rename : taskcluster/docker/debian-base/cloud-mirror-workaround.sh => taskcluster/docker/debian-raw/cloud-mirror-workaround.sh
rename : taskcluster/docker/debian-base/setup_packages.sh => taskcluster/docker/debian-raw/setup_packages.sh
to give time to docker images and toolchains to build.
--HG--
rename : taskcluster/docker/debian-raw/cloud-mirror-workaround.sh => taskcluster/docker/debian-base/cloud-mirror-workaround.sh
rename : taskcluster/docker/debian-raw/setup_packages.sh => taskcluster/docker/debian-base/setup_packages.sh
We however leave moving the packages building to a script for another
day.
Differential Revision: https://phabricator.services.mozilla.com/D19624
--HG--
rename : taskcluster/docker/debian-base/cloud-mirror-workaround.sh => taskcluster/docker/debian-raw/cloud-mirror-workaround.sh
rename : taskcluster/docker/debian-base/setup_packages.sh => taskcluster/docker/debian-raw/setup_packages.sh
We however leave moving the packages building to a script for another
day.
Differential Revision: https://phabricator.services.mozilla.com/D19624
--HG--
rename : taskcluster/docker/debian-base/cloud-mirror-workaround.sh => taskcluster/docker/debian-raw/cloud-mirror-workaround.sh
rename : taskcluster/docker/debian-base/setup_packages.sh => taskcluster/docker/debian-raw/setup_packages.sh
We however leave moving the packages building to a script for another
day.
Differential Revision: https://phabricator.services.mozilla.com/D19624
--HG--
rename : taskcluster/docker/debian-base/cloud-mirror-workaround.sh => taskcluster/docker/debian-raw/cloud-mirror-workaround.sh
rename : taskcluster/docker/debian-base/setup_packages.sh => taskcluster/docker/debian-raw/setup_packages.sh
1. The unsetting of LD_LIBRARY_PATH is removed, because it's no longer
necessary and interferes with environments where it's necessary to find
"system" libraries like GTK; see bug 1472589 comment #1 through #4.
2. The Snap package manifest adds a dependency on the libcurl package,
so that the crash reporter can send the report. This uses the GnuTLS
variant because we're already pulling in GnuTLS as a dependency of some
other packages (FFmpeg and CUPS, but also the non-GnuTLS cURL packages
depend on it anyway via OpenLDAP).
Differential Revision: https://phabricator.services.mozilla.com/D18625
--HG--
extra : moz-landing-system : lando
Ensure the working area on disk is set up for each unique partials generation call, to avoid re-using a broken area in retries.
Differential Revision: https://phabricator.services.mozilla.com/D17928
--HG--
extra : moz-landing-system : lando
***
Bug 1514594: Part 3a - Change ChromeUtils.import to return an exports object; not pollute global. r=mccr8
This changes the behavior of ChromeUtils.import() to return an exports object,
rather than a module global, in all cases except when `null` is passed as a
second argument, and changes the default behavior not to pollute the global
scope with the module's exports. Thus, the following code written for the old
model:
ChromeUtils.import("resource://gre/modules/Services.jsm");
is approximately the same as the following, in the new model:
var {Services} = ChromeUtils.import("resource://gre/modules/Services.jsm");
Since the two behaviors are mutually incompatible, this patch will land with a
scripted rewrite to update all existing callers to use the new model rather
than the old.
***
Bug 1514594: Part 3b - Mass rewrite all JS code to use the new ChromeUtils.import API. rs=Gijs
This was done using the followng script:
https://bitbucket.org/kmaglione/m-c-rewrites/src/tip/processors/cu-import-exports.jsm
***
Bug 1514594: Part 3c - Update ESLint plugin for ChromeUtils.import API changes. r=Standard8
Differential Revision: https://phabricator.services.mozilla.com/D16747
***
Bug 1514594: Part 3d - Remove/fix hundreds of duplicate imports from sync tests. r=Gijs
Differential Revision: https://phabricator.services.mozilla.com/D16748
***
Bug 1514594: Part 3e - Remove no-op ChromeUtils.import() calls. r=Gijs
Differential Revision: https://phabricator.services.mozilla.com/D16749
***
Bug 1514594: Part 3f.1 - Cleanup various test corner cases after mass rewrite. r=Gijs
***
Bug 1514594: Part 3f.2 - Cleanup various non-test corner cases after mass rewrite. r=Gijs
Differential Revision: https://phabricator.services.mozilla.com/D16750
--HG--
extra : rebase_source : 359574ee3064c90f33bf36c2ebe3159a24cc8895
extra : histedit_source : b93c8f42808b1599f9122d7842d2c0b3e656a594%2C64a3a4e3359dc889e2ab2b49461bab9e27fc10a7
This produces the same executables (produced for the same platforms) as
those currently pulled from tooltool (modulo timestamps, maybe changes
since last manifest change, etc.). Unfortunately, as of currently, the
Windows variant needs to be cross-built with mingw because it doesn't
compile without some POSIX APIs that MSVC/Windows SDK don't provide.
One thing that is left out of this change is whether to be completely
accurate with the toolchain cache hash (requiring a large list of files
as resources, and making those built very frequently), whether we'd
rely on manual updates, or if we should go with completely uncached
tasks. This can be left for a followup, the tasks not being hooked up
to be actually used by other tasks yet.
Differential Revision: https://phabricator.services.mozilla.com/D16302
--HG--
extra : moz-landing-system : lando
The nightly partial generation code generates URLs with https:// so they should
be accepted as well.
Differential Revision: https://phabricator.services.mozilla.com/D15311
--HG--
extra : moz-landing-system : lando
We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.
Differential Revision: https://phabricator.services.mozilla.com/D13814
--HG--
extra : moz-landing-system : lando
We now have multiple things we may check out. "vcs" meaning "gecko"
is not obvious. Let's change the terminology to be more specific.
Differential Revision: https://phabricator.services.mozilla.com/D13813
--HG--
extra : moz-landing-system : lando
We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.
Differential Revision: https://phabricator.services.mozilla.com/D13814
--HG--
extra : moz-landing-system : lando
We now have multiple things we may check out. "vcs" meaning "gecko"
is not obvious. Let's change the terminology to be more specific.
Differential Revision: https://phabricator.services.mozilla.com/D13813
--HG--
extra : moz-landing-system : lando
Eventually, workers will provide these variables directly
(https://bugzilla.mozilla.org/show_bug.cgi?id=1460015). But for now, this
ensures that TASKCLUSTER_ROOT_URL is set everywhere in production, and
TASKCLUSTER_PROXY_URL is set wherever the proxy is active.
The taskgraph Taskcluster utils module gets a `get_root_url()` that gets the
root URL for the current run, either from an environment variable in production
or, on the command line, defaulting to https://taskcluster.net for user
convenience. When the production instance's URL changes, we can simply change
that default.
Other changes to use this function are reserved for later commits.
This changes the docker build process propagate TASKCLUSTER_ROOT_URL into the
docker images where necessary (using %ARG), specifically to create URLs for
debian repo paths.
--HG--
extra : rebase_source : 4f50e9d066da62a1887baabd8603844c85a32ee6
extra : source : 5ea6f03f845e49d503f5d0283557f54561c41654
This patch adds a toolchain task for building d8 with customized build settings and uses it in jsshell benchmark tests. A customized image with a debian9-base ('custom-v8') is added by this patch as well and is required to build the tool.
Differential Revision: https://phabricator.services.mozilla.com/D14019
--HG--
extra : moz-landing-system : lando
XZ supports rewritting addresses in executable code, which is architechture
specific. The updater is compiled with support for the target architecture
only, so we can't always compress updates passing `--x86` to XZ. This threads
the architecture through to the repackage steps, so we can pass the appropraite
flags to the update packaging scripts.
Differential Revision: https://phabricator.services.mozilla.com/D14601
--HG--
extra : moz-landing-system : lando
Although this task technically doesn't build a toolchain, the set of
steps it needs to do is very similar to what a toolchain build does, so
we're shoehorning this task into the toolchain kind. The task basically
runs `cargo vendor` on the gfx/wr/Cargo.lock file (if/when it changes)
and exports a tarball of the resulting vendored crates. This allows
downstream tasks that build stuff in gfx/wr to not have to re-fetch
these crates from crates.io on every test run.
Differential Revision: https://phabricator.services.mozilla.com/D14406
--HG--
extra : moz-landing-system : lando
This appears to "just work."
While I would like to convert this image to Debian and make it
deterministic, that is more effect than I'm willing to invest at the
moment.
The impetus for this change is unblocking partial clones. Mercurial's
SQLite storage backend apparently hits a SQLite bug in version 3.11
of SQLite (what Ubuntu 16.04 runs) where SQLite complains about
database corruption when there are readers from multiple processes.
Ubuntu 18.04 is running SQLite 3.22 and doesn't exhibit the buggy
behavior.
Differential Revision: https://phabricator.services.mozilla.com/D14228
--HG--
extra : moz-landing-system : lando
Eventually, workers will provide these variables directly
(https://bugzilla.mozilla.org/show_bug.cgi?id=1460015). But for now, this
ensures that TASKCLUSTER_ROOT_URL is set everywhere, and TASKCLUSTER_PROXY_URL
is set wherever the proxy is active.
The setup for the mach commands defaults to https://taskcluster.net for user
convenience. When the production instance's URL changes, we can simply change
that default.
This changes the docker build process to propagate TASKCLUSTER_ROOT_URL into
the docker images, and for good measure includes some code to use that value to
generate debian repo paths.
Differential Revision: https://phabricator.services.mozilla.com/D14196
--HG--
extra : moz-landing-system : lando
For historical consistency and consistency with index paths.
"env_prefix" is no longer used after this change, so it has been
removed.
Differential Revision: https://phabricator.services.mozilla.com/D13876
--HG--
extra : moz-landing-system : lando
We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.
Differential Revision: https://phabricator.services.mozilla.com/D13814
--HG--
extra : moz-landing-system : lando
We now have multiple things we may check out. "vcs" meaning "firefox"
is not obvious. Let's change the terminology to be more specific.
Differential Revision: https://phabricator.services.mozilla.com/D13813
--HG--
extra : moz-landing-system : lando
This was released a few minutes ago. It contains some fixes necessary to support
partial clones.
Differential Revision: https://phabricator.services.mozilla.com/D13768
--HG--
extra : moz-landing-system : lando
We need to run Mercurial 4.8 so we can start using SQLite storage
and wire protocol version 2 for partial clones.
--auto-deconfigure was added because desktop1604-test now inherits
from a base image with Mercurial installed.
Differential Revision: https://phabricator.services.mozilla.com/D11399
--HG--
extra : moz-landing-system : lando
Counterintuitively, gnome-keyring-daemon needs its capabilities removed in order for it to run in
docker (doing so means that it can't lock secrets in memory, but since this is for tests and we
aren't storing any actually sensitive secrets, this should be fine).
This patch also makes sure gnome-keyring-daemon is running with an unlocked keychain before the
tests are run.
Differential Revision: https://phabricator.services.mozilla.com/D13020
--HG--
extra : moz-landing-system : lando
We need to run Mercurial 4.8 so we can start using SQLite storage
and wire protocol version 2 for partial clones.
Differential Revision: https://phabricator.services.mozilla.com/D11399
--HG--
extra : rebase_source : 712fa08226f74ef9cfe306b7c68ff3dd3f54d9d3
extra : source : a438eff7d106f25a951c690c22f30b37d40d54a6
This uses the latest image_builder image (on docker hub) to build even the
image_builder image.
The change to `docker.py` handles a new API response (`aux`) from the Docker
daemon. It's unclear what this key means, but displaying it is simple.
Differential Revision: https://phabricator.services.mozilla.com/D8441
--HG--
extra : rebase_source : b6a2c2de231bd623521a0a7a0dc595fed059b758
extra : intermediate-source : aa32ba9e6df3e20aab896a63538c8e8a432b4988
extra : source : f33b9e6aca06b7883f048c6c068704680cbfbbb2
The desktop1604-test image doesn't build deterministically, primarily
due to not pinning APT package indices. Rebuilding it from scratch
today results in at least 2 problems causing test failures. One has
to do with a renamed media device. Another is not yet determined but
is causing the Android emulator to fail.
Various people have spent several hours trying to debug the failures.
There are some leads. But no obvious solution is in sight. Various
code landings are blocked on this.
Enough is enough.
This commit modifies desktop1604-test so it inherits from a base image
that contains the last good build of the image. Other minimal
modifications were made to enable the image to build with a populated
base image.
This is a gross hack. It is equivalent to sweeping dust under the rug.
But it gets the job done. It should unblock various code landings.
The base image was imported from task K-qJkvIqQQ-EFK8-4zg_nA.
The intent is to undo this hack ASAP by fixing the base image. This is
tracked by bug 1511527.
As part of developing this patch, I ran into at least two different
issues building the Docker image using the last known good image as a
base.
On my desktop machine using btrfs for Docker storage, the image failed to
build due to a "failed to export image: failed to create image: failed
to get layer XXX: layer does not exist" error.
In CI, I ran into separate issues where some `rm` invocations didn't
work. Why, I have no clue.
I ran a `docker export | docker import` of a running container with
the good image to produce a new Docker image with just the filesystem
contents. This had the effect of squashing the original Docker image
into a single layer. For reasons I can't explain, this made the image
build in CI. Welcome to the wonderful world of Docker bugs, everyone.
Differential Revision: https://phabricator.services.mozilla.com/D13600
--HG--
extra : moz-landing-system : lando
When Content-Encoding is specified, the decoded length won't match the lenght
in the header. In any case aiohttp has code that verifies the length, so we
don't need to do it as well.
Differential Revision: https://phabricator.services.mozilla.com/D12342
--HG--
extra : moz-landing-system : lando
We need to run Mercurial 4.8 so we can start using SQLite storage
and wire protocol version 2 for partial clones.
Differential Revision: https://phabricator.services.mozilla.com/D11399
--HG--
extra : moz-landing-system : lando
This uses the latest image_builder image (on docker hub) to build even the
image_builder image.
The change to `docker.py` handles a new API response (`aux`) from the Docker
daemon. It's unclear what this key means, but displaying it is simple.
Differential Revision: https://phabricator.services.mozilla.com/D8441
--HG--
extra : rebase_source : 2c069da57e416d5e1821e55653d37b23d633ae78
extra : source : f33b9e6aca06b7883f048c6c068704680cbfbbb2
Regenerating the ubuntu1604-test Docker image pulls in a new
package version that changes the name of a media device. This media
device name is currently hard coded in the mochitest test harness.
This commit makes a change to force regeneration of the Docker image
and updates the hardcoded device name to reflect the new version.
Differential Revision: https://phabricator.services.mozilla.com/D11927
--HG--
extra : moz-landing-system : lando
This creates 32-bits variants of the same packages that were added for
64-bits builds, with a few additions:
- python-defaults, so that the python package can be installed as a
dependency of the libglib2.0-dev package,
- xkeyboard-config, so that the xkb-data package can be installed as a
dependency of the libxkbcommon0 package.
Additionally, because the 32-bits and 64-bits packages are built
separately (the 32-bits packages can't, on Wheezy, be built on a 64-bits
host), they don't end up with the same
changelog.Debian/changelog.Debian.gz file because of a timestamp within
it. One way to address this would be to make the taskgraph more complex,
by adding a task creating the source package, and then two tasks
building the 32-bits and 64-bits binary packages from that source, but
that's not worth the overhead, when a simple hack works around the
problem: We make dpkg skip installing the changelog.Debian* files.
Differential Revision: https://phabricator.services.mozilla.com/D11140
If a packaging task ended up being retried for any reason, the downstream
docker tasks that depended on them would fail, since they were hard-coding the
artifacts from the initial run.
Differential Revision: https://phabricator.services.mozilla.com/D7364
This creates 32-bits variants of the same packages that were added for
64-bits builds, with a few additions:
- python-defaults, so that the python package can be installed as a
dependency of the libglib2.0-dev package,
- xkeyboard-config, so that the xkb-data package can be installed as a
dependency of the libxkbcommon0 package.
Additionally, because the 32-bits and 64-bits packages are built
separately (the 32-bits packages can't, on Wheezy, be built on a 64-bits
host), they don't end up with the same
changelog.Debian/changelog.Debian.gz file because of a timestamp within
it. One way to address this would be to make the taskgraph more complex,
by adding a task creating the source package, and then two tasks
building the 32-bits and 64-bits binary packages from that source, but
that's not worth the overhead, when a simple hack works around the
problem: We make dpkg skip installing the changelog.Debian* files.
Differential Revision: https://phabricator.services.mozilla.com/D11140
If a packaging task ended up being retried for any reason, the downstream
docker tasks that depended on them would fail, since they were hard-coding the
artifacts from the initial run.
Differential Revision: https://phabricator.services.mozilla.com/D7364
This creates 32-bits variants of the same packages that were added for
64-bits builds, with a few additions:
- python-defaults, so that the python package can be installed as a
dependency of the libglib2.0-dev package,
- xkeyboard-config, so that the xkb-data package can be installed as a
dependency of the libxkbcommon0 package.
Additionally, because the 32-bits and 64-bits packages are built
separately (the 32-bits packages can't, on Wheezy, be built on a 64-bits
host), they don't end up with the same
changelog.Debian/changelog.Debian.gz file because of a timestamp within
it. One way to address this would be to make the taskgraph more complex,
by adding a task creating the source package, and then two tasks
building the 32-bits and 64-bits binary packages from that source, but
that's not worth the overhead, when a simple hack works around the
problem: We make dpkg skip installing the changelog.Debian* files.
Differential Revision: https://phabricator.services.mozilla.com/D11140
This uses the latest image_builder image (on docker hub) to build even the
image_builder image.
The change to `docker.py` handles a new API response (`aux`) from the Docker
daemon. It's unclear what this key means, but displaying it is simple.
Differential Revision: https://phabricator.services.mozilla.com/D8441
--HG--
extra : moz-landing-system : lando
For tasks that only need python, the base images are sufficient. The in-tree
code verifies that the caches are setup as volumes in the image being used,
so set the caches in the base images as well.
Differential Revision: https://phabricator.services.mozilla.com/D10148
--HG--
extra : moz-landing-system : lando
Updates to periodic_file_updates.sh to complete Thunderbird support
- added comm- branch support so that repository URL's are correctly determined
- set COMMIT_AUTHOR appropriately for Thunderbird
Differential Revision: https://phabricator.services.mozilla.com/D9259
--HG--
extra : moz-landing-system : lando
When a task gets too routes to be indexed by the queue, we create a new
dependent task that generates the index tasks. This task didn't set the index
rank, so if a task gets new routes that push it over the limit, it would stop
updating the old index, as it would be using an index of 0.
Differential Revision: https://phabricator.services.mozilla.com/D10016
--HG--
extra : moz-landing-system : lando
Now autotest does not require java to be installed, but
it will let the user know that infer is not being tested if java
is missing.
Differential Revision: https://phabricator.services.mozilla.com/D7326
--HG--
extra : moz-landing-system : lando
Eventually, workers will provide these variables directly
(https://bugzilla.mozilla.org/show_bug.cgi?id=1460015). But for now, this
ensures that TASKCLUSTER_ROOT_URL is set everywhere, and TASKCLUSTER_PROXY_URL
is set wherever the proxy is active.
The setup for the mach commands defaults to https://taskcluster.net for user
convenience. When the production instance's URL changes, we can simply change
that default.
This changes the docker build process propagate TASKCLUSTER_ROOT_URL into the
docker images where necessary (using %ARG), specifically to create URLs for
debian repo paths.
--HG--
extra : rebase_source : 0626ebdb66a9f4078cb75ab71be256c334297363
The partial generation code checks the URLs of the source versions. To allow
building partials from staging releases, allow the staging CDN when generating
partials.
Differential Revision: https://phabricator.services.mozilla.com/D5710
--HG--
extra : moz-landing-system : lando
pip 18.0.1 and recent pipenv don't work together. Until a new pipenv is released, this is a workaround.
Differential Revision: https://phabricator.services.mozilla.com/D7991
--HG--
extra : moz-landing-system : lando
If a packaging task ended up being retried for any reason, the downstream
docker tasks that depended on them would fail, since they were hard-coding the
artifacts from the initial run.
Differential Revision: https://phabricator.services.mozilla.com/D7364
--HG--
extra : moz-landing-system : lando
This extracts the current logic for finding nodejs into its own module in mozbuild. Configure and ESLint then use it.
For ESLint, this will change the first location it looks for nodejs to be the .mozbuild directory.
Differential Revision: https://phabricator.services.mozilla.com/D6430
--HG--
extra : moz-landing-system : lando
Move all fields of nsISSLStatus to nsITransportSecurityProvider
Remove nsISSLStatus interface and definition
Update all code and test references to nsISSLStatus
Maintain ability to read in older version of serialized nsISSLStatus. This
is verified with psm_DeserializeCert gtest.
Differential Revision: https://phabricator.services.mozilla.com/D3704
--HG--
extra : moz-landing-system : lando
Bumping the Mercurial version to 4.7.1 to apply a fix for a
performance regression introduced in 4.7.
Differential Revision: https://phabricator.services.mozilla.com/D5193
--HG--
extra : rebase_source : b06eae5268fe7f38f85f8644da7f08516840a827
Automatic changes by ESLint, except for manual corrections for .xml files.
Differential Revision: https://phabricator.services.mozilla.com/D4439
--HG--
extra : moz-landing-system : lando
Prior to this patch, getHSTSPreloadList.js would queue an XHR for every preload
list candidate site. This meant that there would be ~50,000 requests in flight
simultaneously. Simply processing these requests caused them to all time out,
and no useful work was done. This patch resolves them in batches of 250 to
avoid this issue.
Differential Revision: https://phabricator.services.mozilla.com/D3622
--HG--
extra : source : e0f0d5824dd8aaaaf1395e569cec1806b028b12e
extra : amend_source : 62f1cd6fc49cbae39c7691c5712906f775862887
Prior to this patch, getHSTSPreloadList.js would queue an XHR for every preload
list candidate site. This meant that there would be ~50,000 requests in flight
simultaneously. Simply processing these requests caused them to all time out,
and no useful work was done. This patch resolves them in batches of 250 to
avoid this issue.
Differential Revision: https://phabricator.services.mozilla.com/D3622
--HG--
extra : moz-landing-system : lando
When the HSTS preload script was reworked to use async/await in bug 1436369,
`fetchstatus` would create an asynchronous xml http request and then attempt to
access a response header from it. However, there was nothing to ensure that the
request had completed before this code ran. This patch ensures that the request
has completed before the response header is used.
This patch also replaces a lingering instance of `Ci.nsISSLStatusProvider` that
should have been changed to `Ci.nsITransportSecurityInfo` in bug 1475647.
Finally, this patch removes the old, redundant getHSTSPreloadList.js in
security/manager/tools as well as the unused nsSTSPreloadList.errors file in
security/manager/ssl.
Differential Revision: https://phabricator.services.mozilla.com/D2807
--HG--
extra : moz-landing-system : lando
This commit points the `install_mercurial.sh` script at the
newest Mercurial distributions uploaded to tooltool.
Differential Revision: https://phabricator.services.mozilla.com/D2626
--HG--
extra : amend_source : a9cd500cc86106750aa5b5472e624a00128cac68
extra : histedit_source : 7a06d9c77b6f7a7b2110dc6bc15858c7aa3b7242
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
This will start to cause an error when a newer shellcheck is available in CI
Differential Revision: https://phabricator.services.mozilla.com/D1940
--HG--
extra : moz-landing-system : lando
1. Updated hgrepo to work with mozilla-beta, mozilla-esr60 and project branches (just in case)
2. Presquashed commits, so we only submit one.
3. Replaced 'which' with 'command -v' to avoid future shellcheck issues.
Differential Revision: https://phabricator.services.mozilla.com/D1582
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : source : 0b941cbdca76fb2fbb98dc5bbc1a0237c69954d0
extra : histedit_source : a3e43bdd8a9a58550bef02fec3be832ca304ea93
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : rebase_source : 4918b8c3bac53d63665006802054038bfbca0314
Let's install python-zstandard for both Python 2 and Python 3 in
all our Debian-based images so it is readily available for use.
MozReview-Commit-ID: 1L8zDc5MYXA
--HG--
extra : rebase_source : db718891dd31d4feceff76fbce753b63049e20b1
Summary:
Attempt to get more information about download timeouts, and
also retry the partial generation if download timeouts happen too often.
Reviewers: mtabara
Reviewed By: mtabara
Bug #: 1452927
Differential Revision: https://phabricator.services.mozilla.com/D1467
--HG--
extra : rebase_source : 5b30ffbde03d6600fecd70452082b674dd3f68d9
Our normal ubuntu 16.04 test image is suitable for hosting an Android x86
emulator with these minor updates: Install kvm and make sure /dev/kvm
rw permissions are open for everyone. Note that /dev/kvm is generally
only visible when running docker with --privileged; its permissions
cannot be modified in the Dockerfile, only at run-time: run-task is the
first opportunity.
download-and-compress isn't very complicated and should work on Python 3
with minimal effort. So let's switch it to use Python 3.
MozReview-Commit-ID: 9G1WfcbbKEY
--HG--
extra : rebase_source : 3a6bab06c8500a90413e8b7642a7bf7bdff04a46
Version 0.9.0 bundles a newer version of the zstandard library, which
is a little faster and has a few minor bug fixes (none that we were
likely hitting, however).
MozReview-Commit-ID: 9YgSZ0G41eg
--HG--
extra : rebase_source : 8f5a68323b1e1fe7e9f1dd1a92e132434972d21d
We want Python 3 available everywhere because it is 2018.
MozReview-Commit-ID: L3wufNXKdnp
--HG--
extra : rebase_source : c260923e3c13f8b28e30eaaf6e1bd38f79500052
The python3-minimal package provides /usr/bin/python3 on Debian.
This commit installs this package so a `python3` executable is
provided.
This required backporting the package to wheezy. The final patch
is trivial. But I wasted a bit of time figuring out why `mk-build-deps`
wasn't working. It would no-op and exit 0 and then the build would
complain about missing dependencies!
glandium's theory is that the ":any" multiarch support on wheezy
isn't complete. Removing ":any" seems to make things "just work."
MozReview-Commit-ID: FBicpK4SmkQ
--HG--
extra : rebase_source : a28ce731024e8ed6a43fb30e2ed57da2abb50d0f
This Dockerfile downloads non-deterministic remote content (by cloning a
Git repo) and then executes code from it. Part of that code is
executing Python package installs.
Since this Docker image was generated, it appears the remote code
requires new build dependencies. This commit adds those package
dependencies.
Not having deterministic Docker image builds is a bug. I'll file a
follow-up so we pin the Git commit used for building so this type
of failure doesn't occur again.
--HG--
extra : amend_source : 533a95abeb7cf7ddc9a1329549f5d294baf983f5
In preparation for making it usable on Windows, after which point
having it in a directory with "docker" in it doesn't make much sense.
MozReview-Commit-ID: Hgu0buFyJwF
--HG--
rename : taskcluster/docker/recipes/run-task => taskcluster/scripts/run-task
extra : rebase_source : 3c0b502d28b5aad54bd04069efbfda88e25bbb20
Previously, we installed the latest version of pip and virtualenv.
This commit pins the pip and virtualenv version so we install known
working versions (pip 10 breaks the image build for some reason).
MozReview-Commit-ID: hOAMencdcr
--HG--
extra : rebase_source : 2cb44c2ef55e29c55cf3d1b354e90d6fb5414cce
