Backed out 3 changesets (bug 1756047, bug 1755516) for causing py3 failures. CLOSED TREE

Backed out changeset f0043e07ec5e (bug 1756047)
Backed out changeset 9fc187cb982e (bug 1755516)
Backed out changeset 5f956232e850 (bug 1755516)
This commit is contained in:
Iulian Moraru 2022-03-10 00:53:18 +02:00
parent 82796b522a
commit 98af6aa00f
5 changed files with 108 additions and 327 deletions

View File

@ -4,6 +4,7 @@ Using third-party Python packages
Mach and its associated commands have a variety of 3rd-party Python dependencies. Many of these Mach and its associated commands have a variety of 3rd-party Python dependencies. Many of these
are vendored in ``third_party/python``, while others are installed at runtime via ``pip``. are vendored in ``third_party/python``, while others are installed at runtime via ``pip``.
Dependencies with "native code" are handled on a machine-by-machine basis.
The dependencies of Mach itself can be found at ``build/mach_virtualenv_packages.txt``. Mach commands The dependencies of Mach itself can be found at ``build/mach_virtualenv_packages.txt``. Mach commands
may have additional dependencies which are specified at ``build/<site>_virtualenv_packages.txt``. may have additional dependencies which are specified at ``build/<site>_virtualenv_packages.txt``.
@ -35,11 +36,6 @@ There's two ways of using 3rd-party Python dependencies:
* :ref:`Vendor the source of the Python package in-tree <python-vendor>`. Dependencies of the Mach * :ref:`Vendor the source of the Python package in-tree <python-vendor>`. Dependencies of the Mach
core logic or of building Firefox itself must be vendored. core logic or of building Firefox itself must be vendored.
.. note::
For dependencies that meet both restrictions (dependency of Mach/build, *and* has
native code), see the :ref:`mach-and-build-native-dependencies` section below.
.. note:: .. note::
If encountering an ``ImportError``, even after following either of the above techniques, If encountering an ``ImportError``, even after following either of the above techniques,
@ -117,105 +113,6 @@ Next, add that package and any new transitive dependencies (you'll see them adde
a Firefox chemspill. Therefore, packages required by Mach core logic or for building a Firefox chemspill. Therefore, packages required by Mach core logic or for building
Firefox itself must be vendored. Firefox itself must be vendored.
.. _mach-and-build-native-dependencies:
Mach/Build Native 3rd-party Dependencies
========================================
There are cases where Firefox is built without being able to ``pip install``, but where
native 3rd party Python dependencies enable optional functionality. This can't be solved
by vendoring the platform-specific libraries, as then each one would have to be stored
multiple times in-tree according to how many platforms we wish to support.
Instead, this is solved by pre-installing such native packages onto the host system
in advance, then having Mach attempt to use such packages directly from the system.
This feature is only viable in very specific environments, as the system Python packages
have to be compatible with Mach's vendored packages.
.. note:
All of these native build-specific dependencies **MUST** be optional requirements
as to support the "no strings attached" builds that only use vendored packages.
To control this behaviour, the ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` environment
variable can be used:
.. list-table:: ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE``
:header-rows: 1
* - ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE``
- Behaviour
* - ``"pip"``
- Mach will ``pip install`` all needed dependencies from PyPI at runtime into a Python
virtual environment that's reused in future Mach invocations.
* - ``"none"``
- Mach will perform the build using only vendored packages. No Python virtual environment
will be created for Mach.
* - ``"system"``
- Mach will use the host system's Python packages as part of doing the build. This option
allows the usage of native Python packages without leaning on a ``pip install`` at
build-time. This is generally slower because the system Python packages have to
be asserted to be compatible with Mach. Additionally, dependency lockfiles are ignored,
so there's higher risk of breakage. Finally, as with ``"none"``, no Python virtualenv
environment is created for Mach.
* - ``<unset>``
- Same behaviour as ``"pip"`` if ``MOZ_AUTOMATION`` isn't set. Otherwise, uses
the same behaviour as ``"system"`` if any needed native Python packages can be found in
the system Python.
There's a couple restrictions here:
* ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` only applies to the top-level ``"mach"`` site,
the ``"common"`` site and the ``"build"`` site. All other sites will use ``pip install`` at
run-time as needed.
* ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"`` is not allowed when using any site other
than ``"mach"``, ``"common"`` or ``"build"``, because:
* As described in :ref:`package-compatibility` below, packages used by Mach are still
in scope when commands are run, and
* The host system is practically guaranteed to be incompatible with commands' dependency
lockfiles.
The ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` environment variable fits into the following use
cases:
Mozilla CI Builds
~~~~~~~~~~~~~~~~~
We need access to the native packages of ``zstandard`` and ``psutil`` to extract archives and
get OS information respectively. Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system"``.
Mozilla CI non-Build Tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~
We generally don't want to create a Mach virtual environment, but it's ok to ``pip install``
for specific command sites as needed. Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``.
Downstream CI Builds
~~~~~~~~~~~~~~~~~~~~
Sometimes these builds happen in sandboxed, network-less environments, and usually these builds
don't need any of the behaviour enabled by installing native Python dependencies.
Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``.
Gentoo Builds
~~~~~~~~~~~~~
When installing Firefox via the package manager, Gentoo generally builds it from source rather than
distributing a compiled binary artifact. Accordingly, users doing a build of Firefox in this
context don't want stray files created in ``~/.mozbuild`` or unnecessary ``pip install`` calls.
Use ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"``.
Firefox Developers
~~~~~~~~~~~~~~~~~~
Leave ``MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE`` unset so that all Mach commands can be run,
Python dependency lockfiles are respected, and optional behaviour is enabled by installing
native packages.
.. _package-compatibility:
Package compatibility Package compatibility
===================== =====================

View File

@ -34,7 +34,7 @@ METADATA_FILENAME = "moz_virtualenv_metadata.json"
# The following virtualenvs *may* be used in a context where they aren't allowed to # The following virtualenvs *may* be used in a context where they aren't allowed to
# install pip packages over the network. In such a case, they must access unvendored # install pip packages over the network. In such a case, they must access unvendored
# python packages via the system environment. # python packages via the system environment.
PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS = ("mach", "build", "common") PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS = ("mach", "build")
class VirtualenvOutOfDateException(Exception): class VirtualenvOutOfDateException(Exception):
@ -50,68 +50,9 @@ class InstallPipRequirementsException(Exception):
class SitePackagesSource(enum.Enum): class SitePackagesSource(enum.Enum):
NONE = "none" NONE = enum.auto()
SYSTEM = "system" SYSTEM = enum.auto()
VENV = "pip" VENV = enum.auto()
@classmethod
def from_environment(cls, external_python, site_name, requirements):
source = os.environ.get("MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE", "").lower()
if source == "system":
source = SitePackagesSource.SYSTEM
elif source == "none":
source = SitePackagesSource.NONE
elif source == "pip":
source = SitePackagesSource.VENV
elif source:
raise Exception(
"Unexpected MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE value, expected one "
'of "system", "pip", "none", or to not be set'
)
if site_name not in PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS:
if source == SitePackagesSource.SYSTEM:
raise Exception(
'Cannot use MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="system" for any '
f"sites other than {PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS}. The "
f'current attempted site is "{site_name}".'
)
return SitePackagesSource.VENV
mach_use_system_python = bool(os.environ.get("MACH_USE_SYSTEM_PYTHON"))
moz_automation = bool(os.environ.get("MOZ_AUTOMATION"))
if source:
if mach_use_system_python:
raise Exception(
"The MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE environment variable is "
"set, so the MACH_USE_SYSTEM_PYTHON variable is redundant and "
"should be unset."
)
return source
if not (mach_use_system_python or moz_automation):
return SitePackagesSource.VENV
# Only print this warning once for the Mach site, so we don't spam it every
# time a site handle is created.
if site_name == "mach" and mach_use_system_python:
print(
'The "MACH_USE_SYSTEM_PYTHON" environment variable is deprecated, '
"please unset it or replace it with either "
'"MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system" or '
'"MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=none"'
)
if not external_python.has_pip():
source = SitePackagesSource.NONE
elif external_python.provides_any_package(site_name, requirements):
source = SitePackagesSource.SYSTEM
else:
source = SitePackagesSource.NONE
return source
class MozSiteMetadata: class MozSiteMetadata:
@ -258,7 +199,7 @@ class MachSiteManager:
def __init__( def __init__(
self, self,
topsrcdir: str, topsrcdir: str,
virtualenv_root: Optional[str], checkout_scoped_state_dir: Optional[str],
requirements: MachEnvRequirements, requirements: MachEnvRequirements,
original_python: "ExternalPythonSite", original_python: "ExternalPythonSite",
site_packages_source: SitePackagesSource, site_packages_source: SitePackagesSource,
@ -266,8 +207,8 @@ class MachSiteManager:
""" """
Args: Args:
topsrcdir: The path to the Firefox repo topsrcdir: The path to the Firefox repo
virtualenv_root: The path to the the associated Mach virtualenv, checkout_scoped_state_dir: The path to the checkout-scoped state_dir,
if any generally ~/.mozbuild/srcdirs/<checkout-based-dir>/
requirements: The requirements associated with the Mach site, parsed from requirements: The requirements associated with the Mach site, parsed from
the file at build/mach_virtualenv_packages.txt the file at build/mach_virtualenv_packages.txt
original_python: The external Python site that was used to invoke Mach. original_python: The external Python site that was used to invoke Mach.
@ -280,7 +221,11 @@ class MachSiteManager:
self._topsrcdir = topsrcdir self._topsrcdir = topsrcdir
self._site_packages_source = site_packages_source self._site_packages_source = site_packages_source
self._requirements = requirements self._requirements = requirements
self._virtualenv_root = virtualenv_root self._virtualenv_root = (
_mach_virtualenv_root(checkout_scoped_state_dir)
if checkout_scoped_state_dir
else None
)
self._metadata = MozSiteMetadata( self._metadata = MozSiteMetadata(
sys.hexversion, sys.hexversion,
"mach", "mach",
@ -320,17 +265,23 @@ class MachSiteManager:
else: else:
original_python = external_python original_python = external_python
source = SitePackagesSource.from_environment( if not _system_python_env_variable_present():
external_python, "mach", requirements source = SitePackagesSource.VENV
) state_dir = get_state_dir()
virtualenv_root = ( elif not external_python.has_pip():
_mach_virtualenv_root(get_state_dir()) source = SitePackagesSource.NONE
if source == SitePackagesSource.VENV state_dir = None
else None else:
) source = (
SitePackagesSource.SYSTEM
if external_python.provides_any_package("mach", requirements)
else SitePackagesSource.NONE
)
state_dir = None
return cls( return cls(
topsrcdir, topsrcdir,
virtualenv_root, state_dir,
requirements, requirements,
original_python, original_python,
source, source,
@ -344,9 +295,7 @@ class MachSiteManager:
*self._requirements.pths_as_absolute(self._topsrcdir), *self._requirements.pths_as_absolute(self._topsrcdir),
*sys.path, *sys.path,
] ]
_assert_pip_check( _assert_pip_check(self._topsrcdir, pthfile_lines, "mach")
self._topsrcdir, pthfile_lines, "mach", self._requirements
)
return True return True
elif self._site_packages_source == SitePackagesSource.VENV: elif self._site_packages_source == SitePackagesSource.VENV:
environment = self._virtualenv() environment = self._virtualenv()
@ -468,7 +417,7 @@ class CommandSiteManager:
def __init__( def __init__(
self, self,
topsrcdir: str, topsrcdir: str,
mach_virtualenv_root: Optional[str], checkout_scoped_state_dir: Optional[str],
virtualenv_root: str, virtualenv_root: str,
site_name: str, site_name: str,
active_metadata: MozSiteMetadata, active_metadata: MozSiteMetadata,
@ -478,7 +427,8 @@ class CommandSiteManager:
""" """
Args: Args:
topsrcdir: The path to the Firefox repo topsrcdir: The path to the Firefox repo
mach_virtualenv_root: The path to the Mach virtualenv, if any checkout_scoped_state_dir: The path to the checkout-scoped state_dir,
generally ~/.mozbuild/srcdirs/<checkout-based-dir>/
virtualenv_root: The path to the virtualenv associated with this site virtualenv_root: The path to the virtualenv associated with this site
site_name: The name of this site, such as "build" site_name: The name of this site, such as "build"
active_metadata: The currently-active moz-managed site active_metadata: The currently-active moz-managed site
@ -488,7 +438,7 @@ class CommandSiteManager:
the file at build/<site_name>_virtualenv_packages.txt the file at build/<site_name>_virtualenv_packages.txt
""" """
self._topsrcdir = topsrcdir self._topsrcdir = topsrcdir
self._mach_virtualenv_root = mach_virtualenv_root self._checkout_scoped_state_dir = checkout_scoped_state_dir
self.virtualenv_root = virtualenv_root self.virtualenv_root = virtualenv_root
self._site_name = site_name self._site_name = site_name
self._virtualenv = PythonVirtualenv(self.virtualenv_root) self._virtualenv = PythonVirtualenv(self.virtualenv_root)
@ -528,27 +478,39 @@ class CommandSiteManager:
active_metadata active_metadata
), "A Mach-managed site must be active before doing work with command sites" ), "A Mach-managed site must be active before doing work with command sites"
external_python = ExternalPythonSite(sys.executable)
requirements = resolve_requirements(topsrcdir, site_name) requirements = resolve_requirements(topsrcdir, site_name)
source = SitePackagesSource.from_environment( if (
external_python, site_name, requirements not _system_python_env_variable_present()
) or site_name not in PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS
if source == SitePackagesSource.NONE and requirements.pypi_requirements: ):
raise Exception( source = SitePackagesSource.VENV
f'The "{site_name}" site requires pip ' else:
"packages, and Mach has been told to find such pip packages in " external_python = ExternalPythonSite(sys.executable)
"the system environment, but it can't because the system doesn't " if not external_python.has_pip():
'have "pip" installed.' if requirements.pypi_requirements:
) raise Exception(
f'The "{site_name}" site requires pip '
"packages, and Mach has been told to find such pip packages in "
"the system environment, but it can't because the system doesn't "
'have "pip" installed.'
)
source = SitePackagesSource.NONE
else:
source = (
SitePackagesSource.SYSTEM
if external_python.provides_any_package(site_name, requirements)
else SitePackagesSource.NONE
)
mach_virtualenv_root = ( checkout_scoped_state_dir = (
_mach_virtualenv_root(get_state_dir()) get_state_dir()
if active_metadata.mach_site_packages_source == SitePackagesSource.VENV if active_metadata.mach_site_packages_source == SitePackagesSource.VENV
else None else None
) )
return cls( return cls(
topsrcdir, topsrcdir,
mach_virtualenv_root, checkout_scoped_state_dir,
os.path.join(command_virtualenvs_dir, site_name), os.path.join(command_virtualenvs_dir, site_name),
site_name, site_name,
active_metadata, active_metadata,
@ -722,9 +684,11 @@ class CommandSiteManager:
lines.extend(system_sys_path) lines.extend(system_sys_path)
elif mach_site_packages_source == SitePackagesSource.VENV: elif mach_site_packages_source == SitePackagesSource.VENV:
# When Mach is using its on-disk virtualenv, add its site-packages directory. # When Mach is using its on-disk virtualenv, add its site-packages directory.
assert self._mach_virtualenv_root assert self._checkout_scoped_state_dir
lines.append( lines.append(
PythonVirtualenv(self._mach_virtualenv_root).site_packages_dir() PythonVirtualenv(
_mach_virtualenv_root(self._checkout_scoped_state_dir)
).site_packages_dir()
) )
# Add this command's vendored and first-party modules. # Add this command's vendored and first-party modules.
@ -758,9 +722,7 @@ class CommandSiteManager:
self._site_packages_source == SitePackagesSource.SYSTEM self._site_packages_source == SitePackagesSource.SYSTEM
or self._mach_site_packages_source == SitePackagesSource.SYSTEM or self._mach_site_packages_source == SitePackagesSource.SYSTEM
): ):
_assert_pip_check( _assert_pip_check(self._topsrcdir, pthfile_lines, self._site_name)
self._topsrcdir, pthfile_lines, self._site_name, self._requirements
)
return _is_venv_up_to_date( return _is_venv_up_to_date(
self._topsrcdir, self._topsrcdir,
@ -882,7 +844,7 @@ class PythonVirtualenv:
return _resolve_installed_packages(self.python_path) return _resolve_installed_packages(self.python_path)
class RequirementsValidationResult: class ExternalSitePackageValidationResult:
def __init__(self): def __init__(self):
self._package_discrepancies = [] self._package_discrepancies = []
self.has_all_packages = True self.has_all_packages = True
@ -902,29 +864,6 @@ class RequirementsValidationResult:
lines.append(f"{requirement}: {error}") lines.append(f"{requirement}: {error}")
return "\n".join(lines) return "\n".join(lines)
@classmethod
def from_packages(cls, packages, requirements):
result = cls()
for pkg in requirements.pypi_requirements:
installed_version = packages.get(pkg.requirement.name)
if not installed_version or not pkg.requirement.specifier.contains(
installed_version
):
result.add_discrepancy(pkg.requirement, installed_version)
elif installed_version:
result.provides_any_package = True
for pkg in requirements.pypi_optional_requirements:
installed_version = packages.get(pkg.requirement.name)
if installed_version and not pkg.requirement.specifier.contains(
installed_version
):
result.add_discrepancy(pkg.requirement, installed_version)
elif installed_version:
result.provides_any_package = True
return result
class ExternalPythonSite: class ExternalPythonSite:
"""Represents the Python site that is executing Mach """Represents the Python site that is executing Mach
@ -966,9 +905,25 @@ class ExternalPythonSite:
def provides_any_package(self, virtualenv_name, requirements): def provides_any_package(self, virtualenv_name, requirements):
system_packages = self._resolve_installed_packages() system_packages = self._resolve_installed_packages()
result = RequirementsValidationResult.from_packages( result = ExternalSitePackageValidationResult()
system_packages, requirements for pkg in requirements.pypi_requirements:
) installed_version = system_packages.get(pkg.requirement.name)
if not installed_version or not pkg.requirement.specifier.contains(
installed_version
):
result.add_discrepancy(pkg.requirement, installed_version)
elif installed_version:
result.provides_any_package = True
for pkg in requirements.pypi_optional_requirements:
installed_version = system_packages.get(pkg.requirement.name)
if installed_version and not pkg.requirement.specifier.contains(
installed_version
):
result.add_discrepancy(pkg.requirement, installed_version)
elif installed_version:
result.provides_any_package = True
if not result.has_all_packages: if not result.has_all_packages:
print(result.report()) print(result.report())
raise Exception( raise Exception(
@ -1021,6 +976,12 @@ def _virtualenv_py_path(topsrcdir):
) )
def _system_python_env_variable_present():
return any(
os.environ.get(var) for var in ("MACH_USE_SYSTEM_PYTHON", "MOZ_AUTOMATION")
)
def _resolve_installed_packages(python_executable): def _resolve_installed_packages(python_executable):
pip_json = subprocess.check_output( pip_json = subprocess.check_output(
[ [
@ -1039,7 +1000,7 @@ def _resolve_installed_packages(python_executable):
return {package["name"]: package["version"] for package in installed_packages} return {package["name"]: package["version"] for package in installed_packages}
def _assert_pip_check(topsrcdir, pthfile_lines, virtualenv_name, requirements): def _assert_pip_check(topsrcdir, pthfile_lines, virtualenv_name):
"""Check if the provided pthfile lines have a package incompatibility """Check if the provided pthfile lines have a package incompatibility
If there's an incompatibility, raise an exception and allow it to bubble up since If there's an incompatibility, raise an exception and allow it to bubble up since
@ -1051,10 +1012,21 @@ def _assert_pip_check(topsrcdir, pthfile_lines, virtualenv_name, requirements):
# Don't re-assert compatibility against the system python within Mach subshells. # Don't re-assert compatibility against the system python within Mach subshells.
return return
print( if (
'Running "pip check" to verify compatibility between the system Python and the ' virtualenv_name == "mach"
f'"{virtualenv_name}" site.' and os.environ.get("MACH_USE_SYSTEM_PYTHON")
) and not os.environ.get("MOZ_AUTOMATION")
):
# Since this assertion takes some time, warn users who have explicitly opted
# in. Since we only want this message to be printed once, only do it for the
# first virtualenv that's used (which is always "mach").
print(
"Since Mach has been requested to use the system Python "
"environment, it will need to verify compatibility before "
"running the current command. This may take a couple seconds.\n"
"Note: you can avoid this delay by unsetting the "
"MACH_USE_SYSTEM_PYTHON environment variable."
)
with tempfile.TemporaryDirectory() as check_env_path: with tempfile.TemporaryDirectory() as check_env_path:
# Pip detects packages on the "sys.path" that have a ".dist-info" or # Pip detects packages on the "sys.path" that have a ".dist-info" or
@ -1084,18 +1056,6 @@ def _assert_pip_check(topsrcdir, pthfile_lines, virtualenv_name, requirements):
f.write("\n".join(pthfile_lines)) f.write("\n".join(pthfile_lines))
pip = [check_env.python_path, "-m", "pip"] pip = [check_env.python_path, "-m", "pip"]
packages = _resolve_installed_packages(check_env.python_path)
validation_result = RequirementsValidationResult.from_packages(
packages, requirements
)
if not validation_result.has_all_packages:
subprocess.check_call(pip + ["list", "-v"], stdout=sys.stderr)
print(validation_result.report(), file=sys.stderr)
raise Exception(
f'The "{virtualenv_name}" site is not compatible with the installed '
"system Python packages."
)
check_result = subprocess.run( check_result = subprocess.run(
pip + ["check"], pip + ["check"],
stdout=subprocess.PIPE, stdout=subprocess.PIPE,
@ -1255,9 +1215,4 @@ def activate_virtualenv(virtualenv: PythonVirtualenv):
def _mach_virtualenv_root(checkout_scoped_state_dir): def _mach_virtualenv_root(checkout_scoped_state_dir):
workspace = os.environ.get("WORKSPACE")
if os.environ.get("MOZ_AUTOMATION") and workspace:
# In CI, put Mach virtualenv in the $WORKSPACE dir, which should be cleaned
# between jobs.
return os.path.join(workspace, "mach_virtualenv")
return os.path.join(checkout_scoped_state_dir, "_virtualenvs", "mach") return os.path.join(checkout_scoped_state_dir, "_virtualenvs", "mach")

View File

@ -10,8 +10,6 @@ from pathlib import Path
from typing import Union from typing import Union
from unittest.mock import Mock from unittest.mock import Mock
from mach.site import MozSiteMetadata, SitePackagesSource
class NoopTelemetry(object): class NoopTelemetry(object):
def __init__(self, failed_glean_import): def __init__(self, failed_glean_import):
@ -22,23 +20,9 @@ class NoopTelemetry(object):
def submit(self, is_bootstrap): def submit(self, is_bootstrap):
if self._failed_glean_import and not is_bootstrap: if self._failed_glean_import and not is_bootstrap:
active_site = MozSiteMetadata.from_runtime()
if active_site.mach_site_packages_source == SitePackagesSource.SYSTEM:
hint = (
"Mach is looking for glean in the system packages. This can be "
"resolved by installing it there, or by allowing Mach to run "
"without using the system Python packages."
)
elif active_site.mach_site_packages_source == SitePackagesSource.NONE:
hint = (
"This is because Mach is currently configured without a source "
"for native Python packages."
)
else:
hint = "You may need to run |mach bootstrap|."
print( print(
f"Glean could not be found, so telemetry will not be reported. {hint}", "Glean could not be found, so telemetry will not be reported. "
"You may need to run |mach bootstrap|.",
file=sys.stderr, file=sys.stderr,
) )

View File

@ -12,7 +12,6 @@ skip-if = python == 3
skip-if = python == 3 skip-if = python == 3
[test_logger.py] [test_logger.py]
[test_mach.py] [test_mach.py]
[test_site.py]
[test_site_activation.py] [test_site_activation.py]
[test_site_compatibility.py] [test_site_compatibility.py]
# The Windows and Mac workers only use the internal PyPI mirror, # The Windows and Mac workers only use the internal PyPI mirror,

View File

@ -1,54 +0,0 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
from __future__ import absolute_import, unicode_literals
import os
from unittest import mock
from unittest.mock import Mock
import pytest as pytest
from mozunit import main
from mach.site import SitePackagesSource
@pytest.mark.parametrize(
"env_native_package_source,env_use_system_python,env_moz_automation,expected",
[
("system", False, False, SitePackagesSource.SYSTEM),
("pip", False, False, SitePackagesSource.VENV),
("none", False, False, SitePackagesSource.NONE),
(None, False, False, SitePackagesSource.VENV),
(None, False, True, SitePackagesSource.SYSTEM),
(None, True, False, SitePackagesSource.SYSTEM),
(None, True, True, SitePackagesSource.SYSTEM),
],
)
def test_resolve_package_source(
env_native_package_source, env_use_system_python, env_moz_automation, expected
):
with mock.patch.dict(
os.environ,
{
"MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE": env_native_package_source or "",
"MACH_USE_SYSTEM_PYTHON": "1" if env_use_system_python else "",
"MOZ_AUTOMATION": "1" if env_moz_automation else "",
},
):
assert SitePackagesSource.from_environment(Mock(), "build", None) == expected
def test_resolve_package_source_always_venv_for_most_sites():
# Only sites in PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS have to be able to function
# using only vendored packages or system packages.
# All others must have an associated virtualenv.
assert (
SitePackagesSource.from_environment(Mock(), "python-test", None)
== SitePackagesSource.VENV
)
if __name__ == "__main__":
main()