mirror of
https://github.com/mozilla/gecko-dev.git
synced 2024-11-25 05:41:12 +00:00
bbfa258584
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically: 1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change. 2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781. 3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed. As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages. With that in mind, this patch does all the following: 1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it. 2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`. 3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`). 4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3). 5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes. This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes: 1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI. 2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code. Differential Revision: https://phabricator.services.mozilla.com/D85916
259 lines
7.5 KiB
Bash
Executable File
259 lines
7.5 KiB
Bash
Executable File
#!/bin/sh
|
|
# This Source Code Form is subject to the terms of the Mozilla Public
|
|
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
|
|
# The beginning of this script is both valid POSIX shell and valid Python,
|
|
# such that the script starts with the shell and is reexecuted with
|
|
# the right Python.
|
|
|
|
# Embeds a shell script inside a Python triple quote. This pattern is valid
|
|
# shell because `''':'`, `':'` and `:` are all equivalent, and `:` is a no-op.
|
|
''':'
|
|
# Commands that are to be run with Python 2.
|
|
py2commands="
|
|
android
|
|
awsy-test
|
|
check-spidermonkey
|
|
cramtest
|
|
crashtest
|
|
devtools-css-db
|
|
firefox-ui-functional
|
|
geckodriver
|
|
geckodriver-test
|
|
hazards
|
|
jsapi-tests
|
|
jsshell-bench
|
|
marionette-test
|
|
jstestbrowser
|
|
jstests
|
|
mochitest
|
|
mozharness
|
|
prettier-format
|
|
raptor
|
|
raptor-test
|
|
reftest
|
|
talos-test
|
|
taskcluster-load-image
|
|
telemetry-tests-client
|
|
test
|
|
tps-build
|
|
visualmetrics
|
|
web-platform-tests
|
|
web-platform-tests-update
|
|
wpt
|
|
wpt-manifest-update
|
|
wpt-metadata-merge
|
|
wpt-metadata-summary
|
|
wpt-serve
|
|
wpt-test-paths
|
|
wpt-unittest
|
|
wpt-update
|
|
"
|
|
|
|
# Commands that are to be run with the system Python 3 instead of the
|
|
# virtualenv.
|
|
nativecmds="
|
|
bootstrap
|
|
create-mach-environment
|
|
"
|
|
|
|
run_py() {
|
|
# Try to run a specific Python interpreter.
|
|
py_executable="$1"
|
|
shift
|
|
if which "$py_executable" > /dev/null
|
|
then
|
|
exec "$py_executable" "$0" "$@"
|
|
else
|
|
echo "This mach command requires $py_executable, which wasn't found on the system!"
|
|
case "$py_executable" in
|
|
python2.7|python3) ;;
|
|
*)
|
|
echo "Consider running 'mach bootstrap' or 'mach create-mach-environment' to create the mach virtualenvs, or set MACH_USE_SYSTEM_PYTHON to use the system Python installation over a virtualenv."
|
|
;;
|
|
esac
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
get_command() {
|
|
# Parse the name of the mach command out of the arguments. This is necessary
|
|
# in the presence of global mach arguments that come before the name of the
|
|
# command, e.g. `mach -v build`. We dispatch to the correct Python
|
|
# interpreter depending on the command.
|
|
while true; do
|
|
case $1 in
|
|
-v|--verbose) shift;;
|
|
-l|--log-file)
|
|
if [ "$#" -lt 2 ]
|
|
then
|
|
echo
|
|
break
|
|
else
|
|
shift 2
|
|
fi
|
|
;;
|
|
--log-interval) shift;;
|
|
--log-no-times) shift;;
|
|
-h) shift;;
|
|
--debug-command) shift;;
|
|
--settings)
|
|
if [ "$#" -lt 2 ]
|
|
then
|
|
echo
|
|
break
|
|
else
|
|
shift 2
|
|
fi
|
|
;;
|
|
# When running `./mach help <command>`, the correct Python for <command>
|
|
# needs to be used.
|
|
help) echo $2; break;;
|
|
# When running `./mach mach-completion /path/to/mach <command>`, the
|
|
# correct Python for <command> needs to be used.
|
|
mach-completion) echo $3; break;;
|
|
"") echo; break;;
|
|
*) echo $1; break;;
|
|
esac
|
|
done
|
|
}
|
|
|
|
state_dir=${MOZBUILD_STATE_PATH:-~/.mozbuild}
|
|
command=$(get_command "$@")
|
|
|
|
# If MACH_USE_SYSTEM_PYTHON or MOZ_AUTOMATION are set, always use the
|
|
# python{2.7,3} executables and not the virtualenv locations.
|
|
if [ -z ${MACH_USE_SYSTEM_PYTHON} ] && [ -z ${MOZ_AUTOMATION} ]
|
|
then
|
|
case "$OSTYPE" in
|
|
cygwin|msys|win32) bin_path=Scripts;;
|
|
*) bin_path=bin;;
|
|
esac
|
|
py2executable=$state_dir/_virtualenvs/mach_py2/$bin_path/python
|
|
py3executable=$state_dir/_virtualenvs/mach/$bin_path/python
|
|
else
|
|
py2executable=python2.7
|
|
py3executable=python3
|
|
fi
|
|
|
|
# Check whether we need to run with the native Python 3 interpreter.
|
|
case " $(echo $nativecmds) " in
|
|
*\ $command\ *)
|
|
run_py python3 "$@"
|
|
;;
|
|
esac
|
|
|
|
# Check for the mach subcommand in the Python 2 commands list and run it
|
|
# with the correct interpreter.
|
|
case " $(echo $py2commands) " in
|
|
*\ $command\ *)
|
|
run_py "$py2executable" "$@"
|
|
;;
|
|
*)
|
|
run_py "$py3executable" "$@"
|
|
;;
|
|
esac
|
|
|
|
# Run Python 3 for everything else.
|
|
run_py "$py3executable" "$@"
|
|
'''
|
|
|
|
from __future__ import absolute_import, print_function, unicode_literals
|
|
|
|
import os
|
|
import sys
|
|
|
|
def ancestors(path):
|
|
while path:
|
|
yield path
|
|
(path, child) = os.path.split(path)
|
|
if child == "":
|
|
break
|
|
|
|
def load_mach(dir_path, mach_path):
|
|
if sys.version_info < (3, 5):
|
|
import imp
|
|
mach_bootstrap = imp.load_source('mach_bootstrap', mach_path)
|
|
else:
|
|
import importlib.util
|
|
spec = importlib.util.spec_from_file_location('mach_bootstrap', mach_path)
|
|
mach_bootstrap = importlib.util.module_from_spec(spec)
|
|
spec.loader.exec_module(mach_bootstrap)
|
|
|
|
return mach_bootstrap.bootstrap(dir_path)
|
|
|
|
|
|
def check_and_get_mach(dir_path):
|
|
bootstrap_paths = (
|
|
'build/mach_bootstrap.py',
|
|
# test package bootstrap
|
|
'tools/mach_bootstrap.py',
|
|
)
|
|
for bootstrap_path in bootstrap_paths:
|
|
mach_path = os.path.join(dir_path, bootstrap_path)
|
|
if os.path.isfile(mach_path):
|
|
return load_mach(dir_path, mach_path)
|
|
return None
|
|
|
|
|
|
def setdefaultenv(key, value):
|
|
"""Compatibility shim to ensure the proper string type is used with
|
|
os.environ for the version of Python being used.
|
|
"""
|
|
encoding = "mbcs" if sys.platform == "win32" else "utf-8"
|
|
|
|
if sys.version_info[0] == 2:
|
|
if isinstance(key, unicode):
|
|
key = key.encode(encoding)
|
|
if isinstance(value, unicode):
|
|
value = value.encode(encoding)
|
|
else:
|
|
if isinstance(key, bytes):
|
|
key = key.decode(encoding)
|
|
if isinstance(value, bytes):
|
|
value = value.decode(encoding)
|
|
|
|
os.environ.setdefault(key, value)
|
|
|
|
|
|
def get_mach():
|
|
# Check whether the current directory is within a mach src or obj dir.
|
|
for dir_path in ancestors(os.getcwd()):
|
|
# If we find a "config.status" and "mozinfo.json" file, we are in the objdir.
|
|
config_status_path = os.path.join(dir_path, 'config.status')
|
|
mozinfo_path = os.path.join(dir_path, 'mozinfo.json')
|
|
if os.path.isfile(config_status_path) and os.path.isfile(mozinfo_path):
|
|
import json
|
|
info = json.load(open(mozinfo_path))
|
|
if 'mozconfig' in info:
|
|
# If the MOZCONFIG environment variable is not already set, set it
|
|
# to the value from mozinfo.json. This will tell the build system
|
|
# to look for a config file at the path in $MOZCONFIG rather than
|
|
# its default locations.
|
|
setdefaultenv('MOZCONFIG', info['mozconfig'])
|
|
|
|
if 'topsrcdir' in info:
|
|
# Continue searching for mach_bootstrap in the source directory.
|
|
dir_path = info['topsrcdir']
|
|
|
|
mach = check_and_get_mach(dir_path)
|
|
if mach:
|
|
return mach
|
|
|
|
# If we didn't find a source path by scanning for a mozinfo.json, check
|
|
# whether the directory containing this script is a source directory. We
|
|
# follow symlinks so mach can be run even if cwd is outside the srcdir.
|
|
return check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
|
|
|
|
def main(args):
|
|
mach = get_mach()
|
|
if not mach:
|
|
print('Could not run mach: No mach source directory found.')
|
|
sys.exit(1)
|
|
sys.exit(mach.run(args))
|
|
|
|
|
|
if __name__ == '__main__':
|
|
main(sys.argv[1:])
|