mirror of
https://github.com/mozilla/gecko-dev.git
synced 2025-02-18 23:15:38 +00:00
15510f9f26
Previously, when using the system Python packages, Mach would reuse the values already existing in the `sys.path`. This had two benefits: 1. We didn't have to do work to calculate which paths the "system Python" specifically referred to. 2. This allowed us to support nested Mach calls (such as `./mach --virtualenv psutil ./mach build`). However, it came with its own headaches, specifically around "consistent imports" and Python subprocesses, such as in the following example 1. Mach runs "using system Python" 2. The "build" site (for example) would be activated. The current `sys.path` is included in the virtualenv's `mach.pth` file so that subprocesses will have access to the same packages as the primary Mach process 3. //Something// adds something to the `sys.path` 4. The build virtualenv is redundantly re-activated (such as due to generic setup code for some module). Though we don't physically re-activate the virtualenv, we //do// check that it's up-to-date to validate assumptions - **and it's not!** It's missing the //something// that was added to the `sys.path`. 5. We can't re-create the build virtualenv because it's already active, so: 6. An error is raised so that unexpected out-of-date-ness doesn't fly under the radar. ----- This patch solves this by calculating the "system Python" paths in a deterministic way. Then, even when using the system Python, the Mach and command sites use define themselves in a consistent way. This means that we can't do `./mach --virtualenv <venv> ./mach ...` to shares `venv`'s packages with the whole new Mach process. Fortunately, this has been replaced with moving such packages into the nested Mach command's requirements definition instead. TL;DR: more consistent, less failures. 👍 ----- One more detail, this time around the `sys_path()` function: Ideally, we could calculate the standard library paths *and* the "system" paths with only one Python subprocess. Unfortunately, that's not the case: * If `-S` isn't provided, then it's tricky to separate the standard library paths from the system paths * If `-S` is provided, then doing `site.getsitepackages()` in a virtualenv returns the *system* site packages, not the virtualenv's. To work around this, we call the external python twice, and in parallel. This allows resolving the information we need while avoiding performance costs. Differential Revision: https://phabricator.services.mozilla.com/D143200
This directory contains common Python code. The basic rule is that if Python code is cross-module (that's "module" in the Mozilla meaning - as in "module ownership") and is MPL-compatible, it should go here. What should not go here: * Vendored python modules (use third_party/python instead) * Python that is not MPL-compatible (see other-licenses/) * Python that has good reason to remain close to its "owning" (Mozilla) module (e.g. it is only being consumed from there). Historical information can be found at https://bugzilla.mozilla.org/show_bug.cgi?id=775243 https://bugzilla.mozilla.org/show_bug.cgi?id=1346025