Consolidate Mach virtualenv management to the front of the
Mach process. This obsoletes `./mach create-mach-environment`
and simplifies the `sh` portion of the top-level `./mach` script.
This helps ensure that the Mach virtualenv doesn't become
out-of-sync and simplifies the mental model of the Mach
virtualenv situation.
Differential Revision: https://phabricator.services.mozilla.com/D120401
It was in `nativecmds` to make finding `pip3` easier.
However, since we're removing the distinction between "regular" and
"native" cmds (Mach will create and activate its venv naturally during
initialization), we need to start pruning the `nativecmds` list.
Fortunately, we can use the stored "external_python" value to decide
where to install "moz-phab" to.
The new behaviour continues to support the following use cases:
* Using Mach with system Python in the `$PATH`.
* Using Mach from within a human-managed virtualenv.
* Either of the above^ in combination with `MACH_USE_SYSTEM_PYTHON`.
Differential Revision: https://phabricator.services.mozilla.com/D120397
We've overloaded "bootstrap" to mean three different things:
* The "standalone bootstrap script": `python/mozboot/bin/bootstrap.py`.
This is to freshly clone a new repo, then run `./mach bootstrap`.
* `./mach bootstrap`: Install necessary dependencies and set up the
system for development.
* "Mach bootstrap": do the in-process initialization work Mach needs
before it can run commands.
By using the term "initialize" instead, perhaps we can remove
ambiguity when discussing Mach.
I'm not attached to the name (or this change at all), but I'm interested
in reviewer thoughts :)
Differential Revision: https://phabricator.services.mozilla.com/D120410
I chose to do this at the level of the outer Python invocation because:
1. `python -m cProfile ...` handles writing the file and some other
details. It's possible to rebuild the functionality -- the tools
are there -- but the APIs are awkward.
2. this allows to profile `mach` internals, instead of just the
invoked command's implementation.
This uses the return code of the `get_command` subshell to transmit
the single bit of information "is the flag present".
The Python-level argument is required in order to have `--help` know
about the option and to avoid the `mach` shell script having to filter
arguments.
Differential Revision: https://phabricator.services.mozilla.com/D116151
Updates `./mach bootstrap` to use `--no-interactive` from global args.
Ensures all bootstrap prompts have a default option.
Differential Revision: https://phabricator.services.mozilla.com/D106814
This code is generally missing tests, but I tried it out on the wpt-sync
where it's used and with this fix it worked (previously it had broken
because the wpt libraries started to assume Py 3 semantics)
Differential Revision: https://phabricator.services.mozilla.com/D107992
Shortly after mach was introduced, bug 840588 turned mach into a small
wrapper that could be copied in a directory part of $PATH, making that
wrapper load the right thing from the right source directory.
Years later, that setup is not really supported, and it's extra
complexity that can lead to unpleasant surprises.
Differential Revision: https://phabricator.services.mozilla.com/D102376
Catch additional cases where both __PYVENV_LAUNCHER__ is set and we invoke a
virtualenv python by unsetting the variable in a prominent location.
This will specifically catch the `./mach bootstrap` => `./mach artifact` case.
Differential Revision: https://phabricator.services.mozilla.com/D101809
It's not 100% clear how long this command will live, but it doesn't look
immediately removable, and it was easy to add python3 support.
Differential Revision: https://phabricator.services.mozilla.com/D94179
This command appears to work just fine under python3 already -- I removed it from the py2 list and ran it with various options, and they all worked fine.
Differential Revision: https://phabricator.services.mozilla.com/D94174
The former is a POSIX shell construct, usually a builtin, and the latter
is an external program that is not guaranteed to be available.
When `which` is not available, a full ./mach build succeeds with this
change.
Differential Revision: https://phabricator.services.mozilla.com/D92864
Minor fix to remoteautomation.py: Increment stdoutlen before any type conversions,
to ensure that it accurately reflects the byte offset in the file.
With this last change, 'mach mochitest' appears to run correctly on Android with Python 3:
switch it over to Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D91586
Since `install-moz-phab` is meant to simplify the moz-phab setup flow,
automatically prompting for Phabricator credentials removes an otherwise
manual step.
Detecting the "console_script" location of a package in a
cross-platform, virtualenv-supporting and "--user"-supporting way is
tough, and the most consistent solution seems to be to list the package
contents of moz-phab and look for the one that seems to be the entry
point.
Differential Revision: https://phabricator.services.mozilla.com/D90642
This is only useful for `mach` commands that we want to run with Python 3 by default, but for which running with Python 2 is still useful. We now have one such command: `python-test`.
In `mach`, switch on the presence of the `MACH_PY2` environment variable. We only want to allow this for `python-test`, so do that sanity check in `mach` as well.
Differential Revision: https://phabricator.services.mozilla.com/D89162
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
