There were a bunch of locations where we were doing path shenanigans
with `requirements.pth/.vendored` items.
There was a bit of complexity because we were specifically making each
`pthfile` line be a relative path to support moving the Firefox
topsrcdir without causing issues.
However, now that we're more intelligent about checking if `pthfile`
lines are up-to-date (and since moving your topsrcdir will still require
re-generating the Mach virtualenv), this behaviour became less useful.
So, generalize `MachEnvRequirements` -> "sys.path lines" logic and
reuse it everywhere.
Differential Revision: https://phabricator.services.mozilla.com/D129693
`tqdm` allows rendering and updating a single-line progress bar, which
is useful for all sorts of different work.
Our first use case for this will be replacing `wget` with a pure-python
downloader.
Source here: https://github.com/tqdm/tqdm
Differential Revision: https://phabricator.services.mozilla.com/D129094
The optional `log_handle` argument was only used by:
* Configure, but the output was always dumped to stdout and *not*
`config.log`. The manual logging _was_ used to handle encoding issues,
but those are likely invalidated by the Python 3 migration.
* `./mach doc`, where we were putting virtualenv setup into stderr,
which seems incorrect. The commit adding it doesn't explain why it's
the case, but I'm guessing it shouldn't be too risky to remove.
Additionally, `log_handle` was used very inconsistently: for example,
running `install_pip_package()` would _not_ use `log_handle`.
So, removing `log_handle` removes a bit of abstraction leakage.
Differential Revision: https://phabricator.services.mozilla.com/D129298
Mach already verifies the Python version during initialization.
Maintaining a second version check can be tough to keep up-to-date, as
we're already seeing due to the obsolete Python 2 check.
Differential Revision: https://phabricator.services.mozilla.com/D129297
Now that are prioritizing system over virtualenv site-packages, the
system `pip` is sometimes being used instead.
This is causing issues when the system pip is set up in a
distro-specific way, such as when "debundled":
https://github.com/pypa/pip/blob/9.0.1/pip/_vendor/__init__.py#L53-L61
However, if we vendor `pip`, `setuptools` and `wheel`, and ensure that
they're prioritized in the `sys.path` before anything is imported from
the system, then we can ensure that we're using a modern `pip` _and_
sidestep system-specific pip weirdness.
Note that `pip-compile`'s `--allow-unsafe` flag is not as dangerous as
it sounds.
There's confusion among maintainers about its origin:
https://github.com/jazzband/pip-tools/issues/522
Additionally, it's going to be enabled by default in a future
`pip-tools` release. So, it's not scary for us to embrace here.
Also, heads up that the "pip outdated warning" no longer needs
to be manually silenced, since pip avoids that code path when
not running from an "installed" context.
Differential Revision: https://phabricator.services.mozilla.com/D127182
The previous behaviour was to:
* Never add a `pthfile` to the Mach virtualenv, and
* Always add Mach's paths to the `sys.path` when Mach initializes
However, this meant that `pip` would needlessly install packages
that already exist in the vendored environment.
Tweak `pth` behaviour so that `pip` behaves more efficiently.
Differential Revision: https://phabricator.services.mozilla.com/D120402
As we leverage the Mach environment more, it becomes increasingly
important that it isn't out-of-date on developer machines.
Add an `up_to_date()` check during Mach initialization.
To minimize the cost to startup, I'm skipping the "pip list" check.
This change required moving `virtualenv` from `mozbuild` to `mach` to
make it available during the early stage of Mach init.
Differential Revision: https://phabricator.services.mozilla.com/D127144
Adding them to the pkg_check_modules alone would add a runtime
dependency on them, which is unwanted (the code using those libraries
goes the extra mile to dynamically lookup their symbols).
So we modify pkg_check_modules to allow to only check for headers for a
given set of packages.
Differential Revision: https://phabricator.services.mozilla.com/D129096
Using `if len(value) and value[0].lower() == ...` is confusing because
it doesn't make it super clear what defaults are. (also, the `lower()`
is not necessary).
We also stop setting value to thin when it was originally cross, because
it prevents special-casing for cross-lto, which we do, and it currently
doesn't actually work.
We also don't need to return a full namespace when lto is disabled (and
we already don't in an existing short-circuiting case), so we can remove
a level of indentation.
Differential Revision: https://phabricator.services.mozilla.com/D129609
Adding them to the pkg_check_modules alone would add a runtime
dependency on them, which is unwanted (the code using those libraries
goes the extra mile to dynamically lookup their symbols).
So we modify pkg_check_modules to allow to only check for headers for a
given set of packages.
Differential Revision: https://phabricator.services.mozilla.com/D129096
There's an existing algorithm to check if a virtualenv's installed
packages are up-to-date with its requirements. This patch
extracts that logic so that, in cases where we can't automatically
download needed pip packages, we can at least assert that the ones
installed to the system Python are sufficient to meet our requirements.
The current only case in which this system-checking logic is applied
is when starting the Mach virtualenv and the `MOZ_AUTOMATION` or
`MACH_USE_SYSTEM_PYTHON` environment variable is set.
Differential Revision: https://phabricator.services.mozilla.com/D122890
Having 3 regular domains to test fission scenarios will help migrating to https-first for DevTools mochitests.
example.org and example.com are both available in http and https, but example.net is only available in http for now.
This patch was created by adding https://example.net to build/pgo/server-locations.txt and then running `./mach python build/pgo/genpgocert.py`
Differential Revision: https://phabricator.services.mozilla.com/D127922
- this function (Windows only) is called when gcda are dumped on disk;
- according to its documentation, it's only useful in case of hard failures, so in a CI context, we don't care;
- it drastically decreases the time in the tests and consequently it avoids timeouts when we use slow disks.
Differential Revision: https://phabricator.services.mozilla.com/D127965
