I think they're remnants from the past that we don't really need anymore.
And they're making things more complicated for some pending work of mine.
Differential Revision: https://phabricator.services.mozilla.com/D89687
VSCode doesn't install by default it's command line integration toolset on MacOS
so w don't have a link to `/usr/local/bin/code` instead we must use the binary from the
`Applications` folder.
Also extens this to other platforms even though it's improbable that we are going to use it
there.
Differential Revision: https://phabricator.services.mozilla.com/D90088
In addition to the existing build telemetry, also gather the stats and
report with Glean. This new telemetry is reported in tandem with the existing
telemetry to allow testing and confidence before a full roll-out.
Additionally, Glean isn't compatible with Python 2, so the new telemetry only runs
on Python 3 mach commands.
Differential Revision: https://phabricator.services.mozilla.com/D83572
This was originally meant to allow `virtualenv`s to use packages from a parent Python environment without having to re-install them. This turned out to not pan out as we would have liked, so we're going another way to solve the same problem. Bug 1660351 walked back a bunch of this logic; this patch deletes the rest of it.
Differential Revision: https://phabricator.services.mozilla.com/D89492
This commit does the following.
- Renames `slashslash` as `dumbComments`. As a result, it now comes before
`emptyLines` in alphabetical ordering, which means that if you apply both
`dumbComments` and `emptyLines`, lines that contain only comments will be
fully removed.
(I contemplated changing the filter ordering to match the order specified,
rather than using alphabetical ordering, but that was more invasive and not
obviously better.)
- Changes `dumbComments` so it only applies if the comment is at the start of
the line (with optional leading whitespace). This is so it can be used with
prefs files, which contain lines like `pref("foo", "https://mozilla.org");`
where the `//` must not be treated as a comment.
Note that `slashslash` wasn't being used anywhere.
Depends on D88240
Differential Revision: https://phabricator.services.mozilla.com/D88242
It's not used, probably because it's pretty strange and hard to imagine using
safely. (Stripping leading and trailing space could be useful, but collapsing
sequences of spaces? Hmm.)
Differential Revision: https://phabricator.services.mozilla.com/D88240
The llvm-strip from clang-11 complains about this file. This file doesn't really interest us anyway -- it's imported from elsewhere -- so just avoid it.
Differential Revision: https://phabricator.services.mozilla.com/D89491
This will be found in a couple places, so we might as well make a helper function. For symmetry put it in the same file where we keep the helper function to locate the `state_dir`.
Differential Revision: https://phabricator.services.mozilla.com/D89156
Bug 1659539 caused the unit tests of this class to suddenly start running on Linux; it failed with a type error that suggests this test has never really properly worked, at least not with the version of Mercurial we're using in CI (`unsupported changeid '0' of type <type 'unicode'>`). The class itself isn't used anywhere besides these tests, so just delete the entire class.
Differential Revision: https://phabricator.services.mozilla.com/D89205
The manifest file hasn't actually done anything since XPT definitions were
moved to the libxul binary, and now just generates warnings in local builes.
Differential Revision: https://phabricator.services.mozilla.com/D89197
The ability to get the path to the Python executable from a given `virtualenv` location is generally useful outside the context of all the extra stuff a `VirtualenvManager` provides, so refactor it out into a lighter-weight helper class.
Differential Revision: https://phabricator.services.mozilla.com/D89175
Pipenv is heavy weight and overkill for the purposes it is being used. We'd like to remove it from the tree and |mach python-test| was one of the last remanining use cases.
Remove the `--python` command-line argument as a result. Users who wish to run unit tests with Python 2 can do `MACH_PY2=1 ./mach python-test ...` or `python2 ./mach python-test ...`.
Also update a few unit tests that would break otherwise in the presence of this change.
There were a couple lines in the `setup.py` for `mozlog` that were problematic for tests and was resulting in errors due to the `mozlog` plugin being loaded by `pytest` more than once. We just delete those lines and bump up the major version number of the package to fix it.
Differential Revision: https://phabricator.services.mozilla.com/D88296
Otherwise `gyp` can choose incorrectly when trying to figure out which Python to use to run its internal scripts, which can manifest as test failures in certain circumstances.
Differential Revision: https://phabricator.services.mozilla.com/D89165
This logic is meant to expose packages from a globally-installed Python to be used by the in-`objdir` `virtualenv`s, so for example we don't have to figure out how to install `zstandard` (or other Python packages with native code that may or may not have prebuilt wheels for any given platform) in those `virtualenv`s. Bug 1660351 augmented that logic to work within the requirements of bug 1660353. This worked mostly, but is causing builds to unconditionally break on Arch Linux, caused a couple test failures, and in general is just introducing other weird behaviors downstream, and issues with the resultant `PYTHONPATH`s are hard to diagnose and fix.
In the long-term we'll have to permanently solve the `zstandard` problem and pave the way for other Python packages with native code as well, but that's not an urgent need.
The ultimate goal is to completely remove `inherit-from-parent-environment`, but we can't do that until bug 1659539 is solved.
Partially reverts bugs 1660351. Entirely reverts bug 1660353, restoring that file to as it was before that patch.
Differential Revision: https://phabricator.services.mozilla.com/D89001
Improves glean performance.
Prior to this change, using Glean adds ~500ms to each `mach` run.
After this change, using Glean adds ~200ms to each `mach` run.
Differential Revision: https://phabricator.services.mozilla.com/D88691
This was always a temporary hack because getting `zstandard` installed into the `objdir` `virtualenv`s was impossible. With the changes made in bug 1656993, this is possible now, so we can remove all this.
Differential Revision: https://phabricator.services.mozilla.com/D87809
The intended behavior of `inherit-from-parent-environment` is that the packages from the parent Python environment are available to the sub-`virtualenv`. The implementation of that behavior thus far has been around "site directories", the idea being that custom (non-stdlib) packages are likely to be installed in the "site directory". The limitation of this approach is that there's no one location, in practice, where packages are installed, and it's hard to enumerate a static list of all those possible locations across all platforms.
This patch circumvents the issue by ignoring the "site directory" question entirely and just looking at `sys.path`. If we're inheriting from the parent environment when creating a `virtualenv`, we just ask the parent Python what its `sys.path` is and configure the `virtualenv`'s `sys.path` on startup.
Differential Revision: https://phabricator.services.mozilla.com/D87808
We `normpath()` the `_root` path when we save it, but the input `path` to `get()` is not necessarily also normalized. Normalizing it prevents unnecessary test failures.
Differential Revision: https://phabricator.services.mozilla.com/D88635
There's a usage of `mozpath.join(...)` found while investigating
wildcard-management. Since the return value isn't used here, and the
parameters here aren't mutated, this function call doesn't do anything.
Differential Revision: https://phabricator.services.mozilla.com/D88508
Bug 1645986 solved the problem for most generated files by moving their
rules to the top-level, but we're going to add rules that will end up in
subdirectories, so we have to solve the same problem again, in the
subdirectories.
Differential Revision: https://phabricator.services.mozilla.com/D88389
There are various problems happening when dealing with the output from
setup.py during virtualenv setup, all of which step from the process
command output not being a unicode string in python.
As this code is still used to setup python2 virtualenv, we need to use
the backwards-compatible universal_newlines=True trick.
Differential Revision: https://phabricator.services.mozilla.com/D88372
This, hopefully, begins to address an ongoing global problem where we have few, if any, insights into the performance of individual build tasks (compilations, calls into Python scripts, etc.) At most we have aggregated statistics about how long tiers last, combined with `sccache` aggregates across the entire build (which don't cover non-compilation tasks). This has a few implications:
1. It's impossible to identify bottlenecks, except by going out of your way to notice and reproduce them. e.g. no one, to my knowledge, was aware that `make_dafsa.py` was a bottleneck until someone happened to notice and report it in bug 1629337. We could have systems that automatically detect this sort of thing, or at least that make it easier to do so than by CTRL-C'ing in the middle of the build several times to try to reproduce the problem.
2. It's impossible to detect regressions, unless the regression is so pronounced and severe that it has an immediate impact on the overall build time and triggers build time alerts.
3. It's impossible to identify that you have *fixed* regressions, except by doing ad-hoc timing measurements by building individual `make` targets. This is error-prone and annoying.
Here we propose a low-friction system wherein individual build tasks log their build own perf info. For now, that's a write to `stdout` consisting of the string `BUILDTASK ` followed by a simple JSON object with a start time, end time, the `argv` of the task, and an additional `"context"` key (I anticipate this could be used to annotate the task with relevant per-task for later aggregation, for example: was this an `sccache` cache hit or not? For now, it's empty everywhere). The build controller then collects this data, validates it, and writes out the entire list of build tasks as a JSON file after the build has completed, similarly to what we already do with `build_resources.json`. We already parse some `make` output to do stuff like tracking when we switch tiers, so this isn't a huge architectural shift or anything.
In my opinion this "should" happen at the build system, or `make`, level, but `make` doesn't expose anything resembling this information to my knowledge, so this has to be implemented outside of `make`. One could implement something like this at the `sccache` level but that doesn't touch anything but C/C++/Rust compilation tasks; an ideal solution would support other generic build tasks. We could also fork `make` to add this feature ourselves, but for several reasons I don't think that's tractable. :)
Of course, this approach has downsides:
1. We depend on parsing the `stdout` of `make`, and processes can unfortunately sometimes trample on each other, leading to data loss for individual build tasks occasionally. This is a necessary limitation of the model to my knowledge, and I don't know that it can be fixed generally. In my testing, not much data tends to be lost usually.
2. Dumping arbitrary data to `stdout` isn't always possible or desirable. If you're not careful about it this can also result in noisier-than-necessary tasks, especially when those tasks are not invoked by a parent process that knows how to handle the special `BUILDTASK` lines.
3. This data is raw enough where aggregation is not completely trivial.
4. This functionality has to be added for any new kind of build task whose performance we'd like to track; it doesn't come "for free" due to not being able to be implemented at the build system level.
5. The data isn't awfully small due to the `argv`'s (at this point, not nearly big enough where we need to be concerned about it IMO, but maybe that will change in the future?)
One can imagine a couple other architectures that could avoid the first two problems, namely: 1) we could use a "real" database that would not dump info to `stdout` and wouldn't lose data, like `sqlite3`; or, 2) we could set up another server, similar to `sccache`, that collects this data from subprocesses and aggregates it, making sure not to lose any along the way. Both of these have enough overhead, in terms of engineering effort or actual impact on latency, where I dont know that they make any sense to even attempt implementing. The remaining continue to be real issues, however.
After this is landed there are a few ways forward. We can start uploading these files as build artifacts in CI to allow us to reason about performance impacts of changes in `central`. We can easily add this functionality to the `sccache` client to start tracking those builds as well. We already have a very simple visualization of build tier timing in `mach resource-usage`; we could join that data against the `BUILDTASK` data to produce a very clear visualization of build bottlenecks, i.e., "why is the `export` tier taking so long", etc.
Differential Revision: https://phabricator.services.mozilla.com/D80284
It can be tricky to find out why artifact-lookup fails, especially since
it can be implicitly affected by environment variables.
Log if a known troublemaker (TASKCLUSTER_ROOT_URL) is set when an
artifact-lookup fails.
Differential Revision: https://phabricator.services.mozilla.com/D88304
On some platforms re-creating the `virtualenv`s can be very time-consuming so we'd like to avoid deleting these `virtualenv`s unnecessarily.
Preserve the existing behavior behind a `-f` flag in case unconditionally wiping the `virtualenv`s is what's needed for any reason.
Differential Revision: https://phabricator.services.mozilla.com/D87668
Bug 1659906 removed `../`s from some generated files paths which turned
up not being supported by the faster make backend. Instead of returning
to those relative paths, just support the topobjdir-relative paths
correctly. The new code is derived from the equivalent code in the
recursive make backend.
Differential Revision: https://phabricator.services.mozilla.com/D88097
This generally happens because people cleverly create custom forks of `mozilla-central` that don't have `git-cinnabar` metadata. This is ALWAYS broken in for artifact builds, but people generally don't know that and the error message isn't informative. Instead, identify when this happens as it happens and suggest an immediate, working alternative.
Differential Revision: https://phabricator.services.mozilla.com/D87923
`glean_sdk` can't currently be installed on Apple Silicon or OpenBSD since Glean can't be built from source. While that issue is being resolved (see bug 1660120), allow this installation to fail.
Differential Revision: https://phabricator.services.mozilla.com/D87667
Bug #985141 added this argument without changing all the callers.
Instead of fixing each caller individually, just allow a value not to be
passed in. This is what the underlying MozbuildObject class does
anyways.
Differential Revision: https://phabricator.services.mozilla.com/D87386
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
