Under some conditions, logging fails with messages like:
```
--- Logging error ---
Traceback (most recent call last):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 1083, in emit
msg = self.format(record)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 927, in format
return fmt.format(record)
File "/Users/glandium/mozilla-unified/python/mach/mach/logging.py", line 156, in format
formatted_msg = record.msg.format(**record.params)
AttributeError: 'LogRecord' object has no attribute 'params'
```
In those cases, the record does have a message, as expected, but it's
preformatted and there are no params in the record at all, rather than
an empty dict.
Differential Revision: https://phabricator.services.mozilla.com/D171408
Convert all other imports of `pytoml` within mozilla-central to imports
of `toml`.
As both libraries patterned their API on that of the Python standard-
library module `json`, this is mostly a straight replace, with two
caveats:
* pytoml's exception messages when loading a file provide the file
name; toml's do not. Some additional machinery has been added or
repurposed in a few places to ensure that the relevant filename is
printed.
* In `python_lockfile.py`, the order of arguments to `dump` needed to
be reversed. (`toml` follows `json` in this; `pytoml` differs.)
This patchset does not remove pytoml from `requirements.in`, as `pytoml`
is still used by `compare-locales`.
Differential Revision: https://phabricator.services.mozilla.com/D164155
On POSIX platforms there can be `lib` and `lib64` for `site-packages`.
On Fedora35 once a virtualenv is created, `lib64` is a symlink of `lib`.
Since we can determine what these paths will be before the virtualenv is
created, `realpath` has different results before/after, which is
problematic for our `pthfile`.
Since we only care about resolving the symlink in the prefix (to keep it
in line with the `topobjdir`, we can simply only `realpath` that and
disregard that there's a symlink between `lib` and `lib64`, which makes
our `pthfile` that's created and expected always the same on all
platforms.
Differential Revision: https://phabricator.services.mozilla.com/D162136
`packaging.version` can handle additional text in the version number (eg: `4.0pre1` and `4.0.2rc1`) that `LooseVersion` cannot.
Differential Revision: https://phabricator.services.mozilla.com/D161756
Brew's Python 3.10 causes `virtualenv==20.7.2` to produce a wonky folder
structure (`$venv/opt/homebrew/lib/python3.10/site-packages`?).
This is likely fixed with newer `virtualenv`, but the simpler workaround
here is to use `venv` instead now that Python 3 is always used.
Adds `python3-venv` to docker image so that tests and debian-based tasks
can leverage it.
Differential Revision: https://phabricator.services.mozilla.com/D144872
Brew's Python 3.10 causes `virtualenv==20.7.2` to produce a wonky folder
structure (`$venv/opt/homebrew/lib/python3.10/site-packages`?).
This is likely fixed with newer `virtualenv`, but the simpler workaround
here is to use `venv` instead now that Python 3 is always used.
Adds `python3-venv` to docker image so that tests and debian-based tasks
can leverage it.
Differential Revision: https://phabricator.services.mozilla.com/D144872
Previously most 'reasons' could only be seen if using a debugger, which
was not helpful when there was a problem in CI.
Depends on D158703
Differential Revision: https://phabricator.services.mozilla.com/D158704
This will resolve issues of drive letter uppercase/lowercase mismatch
causing the venv/site to be considered 'out-of'date'.
Differential Revision: https://phabricator.services.mozilla.com/D158703
Brew's Python 3.10 causes `virtualenv==20.7.2` to produce a wonky folder
structure (`$venv/opt/homebrew/lib/python3.10/site-packages`?).
This is likely fixed with newer `virtualenv`, but the simpler workaround
here is to use `venv` instead now that Python 3 is always used.
Adds `python3-venv` to docker image so that tests and debian-based tasks
can leverage it.
Differential Revision: https://phabricator.services.mozilla.com/D144872
De-duping with `set` causes order to no longer be maintained when the
items are distinct. Use the `OrderedDict` technique to de-dupe
consistently instead.
Differential Revision: https://phabricator.services.mozilla.com/D143604
Verify that activating mach and command sites in various configurations
affects the `sys.path` accurately.
Also verify that the `sys.path` is correct between the "activated
process" and fresh processes spawned from created virtualenvs.
Finally, tweak existing path-management behaviour to be more consistent:
* Don't unnecessarily sort `pths` from requirement definitions - it's
more consistent (and more dependable) to keep the provided order.
* Continue removing unwanted `venv` entries from the `sys.path`, but
don't add them back (such as the `$prefix` entry)
* Always remove `venv` entries from the `sys.path` when the venv isn't
being populated. Though this doesn't affect behaviour, it makes
assertions easier and simplifies the runtime state a bit.
Differential Revision: https://phabricator.services.mozilla.com/D143201
Previously, when using the system Python packages, Mach would reuse the
values already existing in the `sys.path`. This had two benefits:
1. We didn't have to do work to calculate which paths the "system
Python" specifically referred to.
2. This allowed us to support nested Mach calls
(such as `./mach --virtualenv psutil ./mach build`).
However, it came with its own headaches, specifically around "consistent
imports" and Python subprocesses, such as in the following example
1. Mach runs "using system Python"
2. The "build" site (for example) would be activated. The current
`sys.path` is included in the virtualenv's `mach.pth` file so that
subprocesses will have access to the same packages as the primary
Mach process
3. //Something// adds something to the `sys.path`
4. The build virtualenv is redundantly re-activated (such as due to
generic setup code for some module). Though we don't physically
re-activate the virtualenv, we //do// check that it's up-to-date to
validate assumptions - **and it's not!** It's missing the
//something// that was added to the `sys.path`.
5. We can't re-create the build virtualenv because it's already active,
so:
6. An error is raised so that unexpected out-of-date-ness doesn't fly
under the radar.
-----
This patch solves this by calculating the "system Python" paths in a
deterministic way. Then, even when using the system Python, the Mach and
command sites use define themselves in a consistent way.
This means that we can't do `./mach --virtualenv <venv> ./mach ...` to
shares `venv`'s packages with the whole new Mach process. Fortunately,
this has been replaced with moving such packages into the nested Mach
command's requirements definition instead.
TL;DR: more consistent, less failures. 👍
-----
One more detail, this time around the `sys_path()` function:
Ideally, we could calculate the standard library paths *and* the
"system" paths with only one Python subprocess. Unfortunately,
that's not the case:
* If `-S` isn't provided, then it's tricky to separate the standard
library paths from the system paths
* If `-S` is provided, then doing `site.getsitepackages()` in a
virtualenv returns the *system* site packages, not the virtualenv's.
To work around this, we call the external python twice, and in
parallel. This allows resolving the information we need while avoiding
performance costs.
Differential Revision: https://phabricator.services.mozilla.com/D143200
Use `poetry` instead of `pip-compile`, then leverage `pip` to assert
that there's no conflicts between environments.
This enables simplifying `requirements.in`, since `colorama` is now
automatically picked up, as vendoring is no longer linux-py3.6-specific.
`poetry==1.2.0a2` was chosen because it was the first version that would
include `pip`/`setuptools`/`wheel` in the `requirements.txt`, yet it was
also the last version that supports Python 3.6 (as still used in CI in
some spots).
Sorry about the `requirements.txt` noise, that's because poetry doesn't
sort hashes yet [1]. Fortunately, you can be confident in
`requirements.txt` changes by checking that this patch doesn't include
changes to vendored packages themselves (I'm including the results of a
`./mach vendor python`).
[1] https://github.com/python-poetry/poetry/issues/5408
Differential Revision: https://phabricator.services.mozilla.com/D142884
Now that we are gleaning a site packages source in a process-global way
(that is *not* conditional on the site that's being managed), we've
removed an edge case from occurring: the case where Mach would use
`SitePackagesSource.NONE`, but the command site would use
`SitePackagesSource.SYSTEM`. This would've occurred if none of Mach's
dependencies were found in the system Python, but instead once the
command site was initialized, one of *its* dependencies were located.
Since that can no longer happen:
* Command sites that don't populate their VENV will *always* use the
same site packages source as Mach
* `pthfile` generation is simplified (the priority of the system paths
is no longer variable)
* We no longer need to track `site_packages_source` in metadata, since
we can use `mach_site_packages_source` instead.
While here, I moved the "`PIP_NETWORK_INSTALL_RESTRICTED_VIRTUALENVS`
sites shouldn't have `pypi_requirements`" check to a test, since there's
no need for it to happen at runtime.
Differential Revision: https://phabricator.services.mozilla.com/D140668
Simplify Mach's decision-making in CI for where to find native Python
packages: unless `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system` is
provided, always ignore system packages.
Removes optional dependency on system's `pip`, instead leaning on
vendored `pip` copy to see what package versions are installed.
Differential Revision: https://phabricator.services.mozilla.com/D140258
The Mach site is asserted to be compatible with all command sites, but
command sites are deliberately //not// asserted to be compatible with
each other - they're //supposed// to have the flexibility of being
able to be incompatible.
Accordingly, let's fail loudly if code tries to activate from one
command site to another.
Note that, due to the Mach site being a superset of common, it's safe to
activate from "common" to "<other command site>". This is needed for
cases like `./mach python --virtualenv ...`.
This required updating the `mozproxy` tests, who would deliberately
activate the `common` site so they could call a `mozproxy` entry point
script. These tests were fixed by instead invoking `mozproxy` as a
module (`-m`) of the current `python-test` site.
Differential Revision: https://phabricator.services.mozilla.com/D141659
This is needed for a few reasons:
* All mach commands can use virtualenvs, not just `build`-related
commands, so the files don't make sense to be in `build/`.
* When locking is added, more files associated with virtualenvs will be
added, and this will change will ease the related directory structure
setup.
* This removes the need for a redundant "_virtualenv_packages" keyword
as part of the manifest filenames.
Differential Revision: https://phabricator.services.mozilla.com/D140382
ogiorgis