Previously, when using the system Python packages, Mach would reuse the
values already existing in the `sys.path`. This had two benefits:
1. We didn't have to do work to calculate which paths the "system
Python" specifically referred to.
2. This allowed us to support nested Mach calls
(such as `./mach --virtualenv psutil ./mach build`).
However, it came with its own headaches, specifically around "consistent
imports" and Python subprocesses, such as in the following example
1. Mach runs "using system Python"
2. The "build" site (for example) would be activated. The current
`sys.path` is included in the virtualenv's `mach.pth` file so that
subprocesses will have access to the same packages as the primary
Mach process
3. //Something// adds something to the `sys.path`
4. The build virtualenv is redundantly re-activated (such as due to
generic setup code for some module). Though we don't physically
re-activate the virtualenv, we //do// check that it's up-to-date to
validate assumptions - **and it's not!** It's missing the
//something// that was added to the `sys.path`.
5. We can't re-create the build virtualenv because it's already active,
so:
6. An error is raised so that unexpected out-of-date-ness doesn't fly
under the radar.
-----
This patch solves this by calculating the "system Python" paths in a
deterministic way. Then, even when using the system Python, the Mach and
command sites use define themselves in a consistent way.
This means that we can't do `./mach --virtualenv <venv> ./mach ...` to
shares `venv`'s packages with the whole new Mach process. Fortunately,
this has been replaced with moving such packages into the nested Mach
command's requirements definition instead.
TL;DR: more consistent, less failures. 👍
-----
One more detail, this time around the `sys_path()` function:
Ideally, we could calculate the standard library paths *and* the
"system" paths with only one Python subprocess. Unfortunately,
that's not the case:
* If `-S` isn't provided, then it's tricky to separate the standard
library paths from the system paths
* If `-S` is provided, then doing `site.getsitepackages()` in a
virtualenv returns the *system* site packages, not the virtualenv's.
To work around this, we call the external python twice, and in
parallel. This allows resolving the information we need while avoiding
performance costs.
Differential Revision: https://phabricator.services.mozilla.com/D143200
Use `poetry` instead of `pip-compile`, then leverage `pip` to assert
that there's no conflicts between environments.
This enables simplifying `requirements.in`, since `colorama` is now
automatically picked up, as vendoring is no longer linux-py3.6-specific.
`poetry==1.2.0a2` was chosen because it was the first version that would
include `pip`/`setuptools`/`wheel` in the `requirements.txt`, yet it was
also the last version that supports Python 3.6 (as still used in CI in
some spots).
Sorry about the `requirements.txt` noise, that's because poetry doesn't
sort hashes yet [1]. Fortunately, you can be confident in
`requirements.txt` changes by checking that this patch doesn't include
changes to vendored packages themselves (I'm including the results of a
`./mach vendor python`).
[1] https://github.com/python-poetry/poetry/issues/5408
Differential Revision: https://phabricator.services.mozilla.com/D142884
The PGO-specific objdir is meant for the PGO build artifacts.
The python virtualenvs just exist for the host, so there's no need to
scope them. Besides, the existing virtualenv infrastructure doesn't know
how to distinguish between "main 'build' site" and "PGO 'build' site",
which is why it raised this "out-of-date" error.
Differential Revision: https://phabricator.services.mozilla.com/D142373
Now that we are gleaning a site packages source in a process-global way
(that is *not* conditional on the site that's being managed), we've
removed an edge case from occurring: the case where Mach would use
`SitePackagesSource.NONE`, but the command site would use
`SitePackagesSource.SYSTEM`. This would've occurred if none of Mach's
dependencies were found in the system Python, but instead once the
command site was initialized, one of *its* dependencies were located.
Since that can no longer happen:
* Command sites that don't populate their VENV will *always* use the
same site packages source as Mach
* `pthfile` generation is simplified (the priority of the system paths
is no longer variable)
* We no longer need to track `site_packages_source` in metadata, since
we can use `mach_site_packages_source` instead.
While here, I moved the "`PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS`
sites shouldn't have `pypi_requirements`" check to a test, since there's
no need for it to happen at runtime.
Differential Revision: https://phabricator.services.mozilla.com/D140668
Simplify Mach's decision-making in CI for where to find native Python
packages: unless `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system` is
provided, always ignore system packages.
Removes optional dependency on system's `pip`, instead leaning on
vendored `pip` copy to see what package versions are installed.
Differential Revision: https://phabricator.services.mozilla.com/D140258
Unfortunately, the dependencies of l10nregistry-rs require tokio 1.0. This
allows this third duplicated instance. I think it makes more sense to try to
move the other uses in tree to a more modern version than to try to downgrade
the version of tokio required by l10nregistry-rs and dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D139854
The dataReportingNotification is the last place where the .properties
vendorShortName is used. As notifications already support Fluent,
porting this to use Fluent as well seems pretty strqaightforward.
Differential Revision: https://phabricator.services.mozilla.com/D141860
The Mach site is asserted to be compatible with all command sites, but
command sites are deliberately //not// asserted to be compatible with
each other - they're //supposed// to have the flexibility of being
able to be incompatible.
Accordingly, let's fail loudly if code tries to activate from one
command site to another.
Note that, due to the Mach site being a superset of common, it's safe to
activate from "common" to "<other command site>". This is needed for
cases like `./mach python --virtualenv ...`.
This required updating the `mozproxy` tests, who would deliberately
activate the `common` site so they could call a `mozproxy` entry point
script. These tests were fixed by instead invoking `mozproxy` as a
module (`-m`) of the current `python-test` site.
Differential Revision: https://phabricator.services.mozilla.com/D141659
This is needed for a few reasons:
* All mach commands can use virtualenvs, not just `build`-related
commands, so the files don't make sense to be in `build/`.
* When locking is added, more files associated with virtualenvs will be
added, and this will change will ease the related directory structure
setup.
* This removes the need for a redundant "_virtualenv_packages" keyword
as part of the manifest filenames.
Differential Revision: https://phabricator.services.mozilla.com/D140382
The intention is to remove all the files in the vendoring directory
(which may be the yaml directory or may not be) except those
specified in 'keep'.
However convert_patterns_to_paths expects a list of patterns, and
we were providing it with only one. This resulted in it iterating
over '**' as a string (one iteration for '*' and a second for '*')
This prevented it from traversing recursively. Turning it into a
list fixes it.
Depends on D141901
Differential Revision: https://phabricator.services.mozilla.com/D141902
This creates a way to override the 'tracking: tag' mechnaism
of a moz.yaml file without having to edit it. Useful to
test a library update when there is no new tag available.
Differential Revision: https://phabricator.services.mozilla.com/D141901
Update DSN for reporting exceptions to the new Sentry instance, and
update CI job to create releases on the new Sentry as well.
Differential Revision: https://phabricator.services.mozilla.com/D141122
Since bug 1717104 changed `./mach vendor python` to no longer have a
manual virtualenv activation, we can remove the test mocking that
softly handled the activation.
Differential Revision: https://phabricator.services.mozilla.com/D141674
All commands declaring a virtualenv will have them activated before the
command executes. Removes all now-redundant manual activations of
declared virtualenvs.
Commands that don't declare a virtualenv will still implicitly be
associated with the "common" virtualenv, but unlike explicit
virtualenv declarations it'll have to be activated manually, just
like it was before this patch.
To smooth the migration with existing usages, virtualenv activation
behaviour was changed slightly: if attempting to activate a new
virtualenv, but the source venv is already command venv, then raise an
exception. (In the future, we should improve testability of
virtualenv scaffolding logic so that tests can be added for this
sort of thing.) This did cause some issues with some tests, which
will be solved more cleanly with bug 1724273. In the meantime,
minimal modifications were made to failing tests to keep them green:
* `test_command_line.py` was activating the `common` virtualenv so
that it could install `mozproxy`, and use its CLI. Instead, I
modified the test to use `mozproxy` using the "module" interface
(`python -m mozproxy ...`). At that point, `MozbuildObject` was
unnecessary and usages were replaced with simpler variants.
* `test_vendor.py` needed its explicit `activate_virtualenv()` call
patched out. It still needs to use a virtualenv's Python
executable, but due to `sys.executable` now being kept up-to-date
as of bug 1717051, it could be used directly.
Differential Revision: https://phabricator.services.mozilla.com/D122892
Creates/updates virtualenvs for some mach commands, replacing
their ad-hoc usage of `install_pip_package()`, `pip install`,
and `sys.path` modifications.
Note: The `docs` virtualenv has `Sphinx==1.1.3` installed, even
though a more modern version of `Sphinx` is used when
`./mach doc` is run. This is ok for now, since `./mach doc` will
just install the newer `Sphinx` over top of the old one. Secondarily,
when we port `./mach doc` to use the centralized system, we'll
be incentivized to make the different `doc` commands use synchonized
versions of the same packages. Success!
Also, note that manual installation of `html5lib` and `requests`
isn't ported to the `wpt` site: this is because they're already
provided by the inherited Mach site.
Differential Revision: https://phabricator.services.mozilla.com/D122902
There are cases in CI where sites are being considered out-of-date,
despite the situation being unexpected - for example, the
"site out-of-date even though already activated" issue described in
this patch's associated bug.
Return a `reason` string from `_is_venv_up_to_date()`, and print it in
cases where a virtualenv being out-of-date is erroneous.
Differential Revision: https://phabricator.services.mozilla.com/D140855
The Mach site management has been significantly simplified
in `mach_initialize.py`, so less of the `MachSiteManager`
API is used. Privatize such now-externally-unused functions,
and no longer return unneeded values.
Differential Revision: https://phabricator.services.mozilla.com/D140854
There are cases in CI where sites are being considered out-of-date,
despite the situation being unexpected - for example, the
"site out-of-date even though already activated" issue described in
this patch's associated bug.
Return a `reason` string from `_is_venv_up_to_date()`, and print it in
cases where a virtualenv being out-of-date is erroneous.
Differential Revision: https://phabricator.services.mozilla.com/D140855
The Mach site management has been significantly simplified
in `mach_initialize.py`, so less of the `MachSiteManager`
API is used. Privatize such now-externally-unused functions,
and no longer return unneeded values.
Differential Revision: https://phabricator.services.mozilla.com/D140854
Previously, when trying to determine the minimal `sys.path` that just
includes the standard library (but no `pip`-installed packages), we
would resolve the `sys.path` and remove the system and user
site-packages. However, this "removal" step didn't work as-is for
`brew`'s Python, because its "system site-packages" is //different//
from its `site.getsitepackages()`.
Stepping back though, there's an easier solution: run `python` with the
`-S` flag, and no "site" paths or custom initialization logic will add
anything extra to the stdlib.
Differential Revision: https://phabricator.services.mozilla.com/D141508
Now that we're using `sysconfig` instead of `distutils`, our `purelib`
and `platlib` paths should correctly point to python environment paths
without a spurious `local/` component at their roots.
Differential Revision: https://phabricator.services.mozilla.com/D140871
Depending on the subprocesses created (such as if
`--with-ccache=sccache` is set), `./mach configure` may hang.
More details about why this is is documented in bug 1753797.
The workaround is to lean on the standard library to stream-and-format
output, rather than our existing `ProcessHandler` code.
Though this still streams both `stdout` and `stderr` in real-time, I saw
some ordering differences between the two streams locally. I don't yet
have a reason to believe that these differences are harmful or otherwise
incorrect.
Differential Revision: https://phabricator.services.mozilla.com/D141028
On newer systems, Python is intelligently populating pure-python and
platform-specific packages into //two different directories//.
Traditionally, these directories would be symlinked - or only one would
exist. However, to support these newer systems (and be better integrated
with Python conventions), we should handle both of these directories
properly.
The solution here is, when calculating the `site_packages_dir()`,
calculate all of them, and handle the multitude at all call sites.
The one use case where we want only one path (the resolving the location
for the `mach.pth` pthfile), I've opted for the `platlib`, since both
pure python dependencies //and// platform-specific dependencies are
referenced by the pthfile.
Differential Revision: https://phabricator.services.mozilla.com/D140870
I'm guessing that the reason we haven't been building the Mach
virtualenv in CI so far is because:
* The default location in `~/.mozbuild` may not be cleared between jobs
* There's a performance cost to `pip install`-ing packages, especially
if they're not needed
* In the past, it was tricky to have some jobs use a Mach virtualenv and
others not.
This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".
This patch addresses the first point about the Mach virtualenv location:
In CI, if `WORKSPACE` is set, the Mach venv will be created in
`$WORKSPACE/mach_virtualenv`.
The other two points are solved by explicit configuration using the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
Differential Revision: https://phabricator.services.mozilla.com/D140256
There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the `"common"`
virtualenv), whose associated `python` binary is used to invoke
Mozharness.
So, just like for the `build` site, allow creating the `common`
site even when the "native package source" is the system Python.
Since `mach_virtualenv_requirements.txt` already depends on
`common_virtualenv_requirements.txt`, this restriction should not cause
validation breakage.
Differential Revision: https://phabricator.services.mozilla.com/D140255
Adds support and documentation for the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the `"system"` where
needed, then remove the backwards-compatibility shim afterwards.
A restriction added by this change is that it's no longer possible to
have the combination of:
* Have Mach use the system Python, and
* Have the active command site do `pip install`.
This is because this case shouldn't be necessary (if `pip install` is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.
When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"`)
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)
Differential Revision: https://phabricator.services.mozilla.com/D138932
I'm guessing that the reason we haven't been building the Mach
virtualenv in CI so far is because:
* The default location in `~/.mozbuild` may not be cleared between jobs
* There's a performance cost to `pip install`-ing packages, especially
if they're not needed
* In the past, it was tricky to have some jobs use a Mach virtualenv and
others not.
This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".
This patch addresses the first point about the Mach virtualenv location:
In CI, if `WORKSPACE` is set, the Mach venv will be created in
`$WORKSPACE/mach_virtualenv`.
The other two points are solved by explicit configuration using the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
Differential Revision: https://phabricator.services.mozilla.com/D140256
There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the `"common"`
virtualenv), whose associated `python` binary is used to invoke
Mozharness.
So, just like for the `build` site, allow creating the `common`
site even when the "native package source" is the system Python.
Since `mach_virtualenv_requirements.txt` already depends on
`common_virtualenv_requirements.txt`, this restriction should not cause
validation breakage.
Differential Revision: https://phabricator.services.mozilla.com/D140255
Adds support and documentation for the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the `"system"` where
needed, then remove the backwards-compatibility shim afterwards.
A restriction added by this change is that it's no longer possible to
have the combination of:
* Have Mach use the system Python, and
* Have the active command site do `pip install`.
This is because this case shouldn't be necessary (if `pip install` is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.
When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"`)
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)
Differential Revision: https://phabricator.services.mozilla.com/D138932