mirror of
https://github.com/mozilla/gecko-dev.git
synced 2024-10-19 16:25:38 +00:00
bbfa258584
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically: 1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change. 2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781. 3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed. As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages. With that in mind, this patch does all the following: 1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it. 2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`. 3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`). 4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3). 5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes. This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes: 1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI. 2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code. Differential Revision: https://phabricator.services.mozilla.com/D85916
113 lines
4.5 KiB
Plaintext
113 lines
4.5 KiB
Plaintext
mozilla.pth:python/mach
|
|
mozilla.pth:python/mozboot
|
|
mozilla.pth:python/mozbuild
|
|
mozilla.pth:python/mozlint
|
|
mozilla.pth:python/mozperftest
|
|
mozilla.pth:python/mozrelease
|
|
mozilla.pth:python/mozterm
|
|
mozilla.pth:python/mozversioncontrol
|
|
mozilla.pth:python/l10n
|
|
mozilla.pth:third_party/python/appdirs
|
|
mozilla.pth:third_party/python/atomicwrites
|
|
mozilla.pth:third_party/python/attrs/src
|
|
python2:mozilla.pth:third_party/python/backports
|
|
mozilla.pth:third_party/python/biplist
|
|
mozilla.pth:third_party/python/blessings
|
|
mozilla.pth:third_party/python/Click
|
|
mozilla.pth:third_party/python/compare-locales
|
|
mozilla.pth:third_party/python/configobj
|
|
mozilla.pth:third_party/python/cookies
|
|
mozilla.pth:third_party/python/cram
|
|
mozilla.pth:third_party/python/diskcache
|
|
mozilla.pth:third_party/python/distro
|
|
mozilla.pth:third_party/python/dlmanager
|
|
mozilla.pth:third_party/python/ecdsa/src
|
|
python2:mozilla.pth:third_party/python/enum34
|
|
mozilla.pth:third_party/python/esprima
|
|
mozilla.pth:third_party/python/fluent.migrate
|
|
mozilla.pth:third_party/python/fluent.syntax
|
|
mozilla.pth:third_party/python/funcsigs
|
|
python2:mozilla.pth:third_party/python/futures
|
|
mozilla.pth:third_party/python/importlib_metadata
|
|
mozilla.pth:third_party/python/iso8601
|
|
mozilla.pth:third_party/python/Jinja2/src
|
|
mozilla.pth:third_party/python/jsonschema
|
|
mozilla.pth:third_party/python/MarkupSafe/src
|
|
mozilla.pth:third_party/python/mohawk
|
|
mozilla.pth:third_party/python/more-itertools
|
|
mozilla.pth:third_party/python/mozilla-version
|
|
mozilla.pth:third_party/python/pathlib2
|
|
mozilla.pth:third_party/python/pathspec
|
|
mozilla.pth:third_party/python/pep487/lib
|
|
mozilla.pth:third_party/python/gyp/pylib
|
|
mozilla.pth:third_party/python/pyrsistent
|
|
mozilla.pth:third_party/python/python-hglib
|
|
mozilla.pth:third_party/python/pluggy
|
|
mozilla.pth:third_party/python/jsmin
|
|
!windows:optional:setup.py:third_party/python/psutil:build_ext:--inplace
|
|
!windows:mozilla.pth:third_party/python/psutil
|
|
windows:mozilla.pth:third_party/python/psutil-cp27-none-win_amd64
|
|
mozilla.pth:third_party/python/pylru
|
|
mozilla.pth:third_party/python/pystache
|
|
python2:mozilla.pth:third_party/python/PyYAML/lib
|
|
python3:mozilla.pth:third_party/python/PyYAML/lib3/
|
|
mozilla.pth:third_party/python/requests
|
|
mozilla.pth:third_party/python/requests-unixsocket
|
|
python2:mozilla.pth:third_party/python/scandir
|
|
mozilla.pth:third_party/python/slugid
|
|
mozilla.pth:third_party/python/taskcluster
|
|
mozilla.pth:third_party/python/taskcluster-urls
|
|
mozilla.pth:third_party/python/py
|
|
mozilla.pth:third_party/python/pytest/src
|
|
mozilla.pth:third_party/python/pytoml
|
|
mozilla.pth:third_party/python/redo
|
|
mozilla.pth:third_party/python/responses
|
|
mozilla.pth:third_party/python/sentry-sdk
|
|
mozilla.pth:third_party/python/six
|
|
mozilla.pth:third_party/python/taskcluster-urls
|
|
mozilla.pth:third_party/python/urllib3/src
|
|
mozilla.pth:third_party/python/voluptuous
|
|
mozilla.pth:third_party/python/json-e
|
|
mozilla.pth:third_party/python/yamllint
|
|
mozilla.pth:third_party/python/zipp
|
|
mozilla.pth:build
|
|
mozilla.pth:config
|
|
mozilla.pth:config/mozunit
|
|
mozilla.pth:dom/bindings
|
|
mozilla.pth:dom/bindings/parser
|
|
mozilla.pth:layout/tools/reftest
|
|
mozilla.pth:third_party/python/ply
|
|
mozilla.pth:taskcluster
|
|
mozilla.pth:testing
|
|
mozilla.pth:testing/condprofile
|
|
mozilla.pth:testing/firefox-ui/harness
|
|
mozilla.pth:testing/marionette/client
|
|
mozilla.pth:testing/marionette/harness
|
|
mozilla.pth:testing/marionette/harness/marionette_harness/runner/mixins/browsermob-proxy-py
|
|
mozilla.pth:testing/marionette/puppeteer/firefox
|
|
mozilla.pth:testing/raptor
|
|
mozilla.pth:testing/talos
|
|
packages.txt:testing/mozbase/packages.txt
|
|
mozilla.pth:tools
|
|
mozilla.pth:testing/web-platform
|
|
mozilla.pth:testing/web-platform/tests/tools/third_party/html5lib
|
|
mozilla.pth:testing/web-platform/tests/tools/third_party/webencodings
|
|
mozilla.pth:testing/web-platform/tests/tools/third_party/h2
|
|
mozilla.pth:testing/web-platform/tests/tools/third_party/hpack
|
|
mozilla.pth:testing/web-platform/tests/tools/third_party/hyperframe
|
|
mozilla.pth:testing/web-platform/tests/tools/third_party/certifi
|
|
mozilla.pth:testing/web-platform/tests/tools/wptserve
|
|
mozilla.pth:testing/web-platform/tests/tools/wptrunner
|
|
mozilla.pth:testing/web-platform/tests/tools/six
|
|
mozilla.pth:testing/xpcshell
|
|
mozilla.pth:third_party/python/mock-1.0.0
|
|
mozilla.pth:xpcom/ds/tools
|
|
mozilla.pth:tools/moztreedocs
|
|
mozilla.pth:third_party/python/cbor2
|
|
mozilla.pth:third_party/python/pyasn1
|
|
mozilla.pth:third_party/python/pyasn1-modules
|
|
mozilla.pth:third_party/python/rsa
|
|
mozilla.pth:toolkit/components/telemetry/tests/marionette/harness
|
|
mozilla.pth:xpcom/idl-parser
|
|
optional:packages.txt:comm/build/virtualenv_packages.txt
|