Now that we're using `sysconfig` instead of `distutils`, our `purelib`
and `platlib` paths should correctly point to python environment paths
without a spurious `local/` component at their roots.
Differential Revision: https://phabricator.services.mozilla.com/D140871
Depending on the subprocesses created (such as if
`--with-ccache=sccache` is set), `./mach configure` may hang.
More details about why this is is documented in bug 1753797.
The workaround is to lean on the standard library to stream-and-format
output, rather than our existing `ProcessHandler` code.
Though this still streams both `stdout` and `stderr` in real-time, I saw
some ordering differences between the two streams locally. I don't yet
have a reason to believe that these differences are harmful or otherwise
incorrect.
Differential Revision: https://phabricator.services.mozilla.com/D141028
On newer systems, Python is intelligently populating pure-python and
platform-specific packages into //two different directories//.
Traditionally, these directories would be symlinked - or only one would
exist. However, to support these newer systems (and be better integrated
with Python conventions), we should handle both of these directories
properly.
The solution here is, when calculating the `site_packages_dir()`,
calculate all of them, and handle the multitude at all call sites.
The one use case where we want only one path (the resolving the location
for the `mach.pth` pthfile), I've opted for the `platlib`, since both
pure python dependencies //and// platform-specific dependencies are
referenced by the pthfile.
Differential Revision: https://phabricator.services.mozilla.com/D140870
I'm guessing that the reason we haven't been building the Mach
virtualenv in CI so far is because:
* The default location in `~/.mozbuild` may not be cleared between jobs
* There's a performance cost to `pip install`-ing packages, especially
if they're not needed
* In the past, it was tricky to have some jobs use a Mach virtualenv and
others not.
This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".
This patch addresses the first point about the Mach virtualenv location:
In CI, if `WORKSPACE` is set, the Mach venv will be created in
`$WORKSPACE/mach_virtualenv`.
The other two points are solved by explicit configuration using the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
Differential Revision: https://phabricator.services.mozilla.com/D140256
There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the `"common"`
virtualenv), whose associated `python` binary is used to invoke
Mozharness.
So, just like for the `build` site, allow creating the `common`
site even when the "native package source" is the system Python.
Since `mach_virtualenv_requirements.txt` already depends on
`common_virtualenv_requirements.txt`, this restriction should not cause
validation breakage.
Differential Revision: https://phabricator.services.mozilla.com/D140255
Adds support and documentation for the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the `"system"` where
needed, then remove the backwards-compatibility shim afterwards.
A restriction added by this change is that it's no longer possible to
have the combination of:
* Have Mach use the system Python, and
* Have the active command site do `pip install`.
This is because this case shouldn't be necessary (if `pip install` is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.
When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"`)
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)
Differential Revision: https://phabricator.services.mozilla.com/D138932
I'm guessing that the reason we haven't been building the Mach
virtualenv in CI so far is because:
* The default location in `~/.mozbuild` may not be cleared between jobs
* There's a performance cost to `pip install`-ing packages, especially
if they're not needed
* In the past, it was tricky to have some jobs use a Mach virtualenv and
others not.
This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".
This patch addresses the first point about the Mach virtualenv location:
In CI, if `WORKSPACE` is set, the Mach venv will be created in
`$WORKSPACE/mach_virtualenv`.
The other two points are solved by explicit configuration using the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
Differential Revision: https://phabricator.services.mozilla.com/D140256
There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the `"common"`
virtualenv), whose associated `python` binary is used to invoke
Mozharness.
So, just like for the `build` site, allow creating the `common`
site even when the "native package source" is the system Python.
Since `mach_virtualenv_requirements.txt` already depends on
`common_virtualenv_requirements.txt`, this restriction should not cause
validation breakage.
Differential Revision: https://phabricator.services.mozilla.com/D140255
Adds support and documentation for the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the `"system"` where
needed, then remove the backwards-compatibility shim afterwards.
A restriction added by this change is that it's no longer possible to
have the combination of:
* Have Mach use the system Python, and
* Have the active command site do `pip install`.
This is because this case shouldn't be necessary (if `pip install` is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.
When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"`)
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)
Differential Revision: https://phabricator.services.mozilla.com/D138932
We want `_pthfile_lines()` to consistently and accurately represent a
virtualenv's `sys.path` modifications.
However, this becomes tricky when comparing expected `pthfile_lines`
//before// activating a virtualenv versus //afterwards//, especially
since Python implicitly adds some paths (such as the path to
`site-packages`).
The current solution made this scrubbing only happen if we were
`pip install`-ing into the site. Unfortunately, it //doesn't// work for
the "use system Python for the `build` site", because:
* Pre-activation result: `site-packages` isn't added, because we aren't
using it
* Post-activation result, implicitly-added `site-packages` isn't
scrubbed, because **scrubbing only happened for
`SitePackagesSource.VENV`**
Stepping back, the solution here is:
* `pthfile_lines` only represents Mach modifications //on top// of
implicit virtualenv behaviour: since `site-packages` is always added
by default, it shouldn't be in our explicit `pthfile_lines`.
* The only time when implicit `sys.path` additions throws us off is when
`SitePackagesSource.SYSTEM`: so, move the scrubbing to happen in that
case instead.
Finally refactor the "conditional deprioritization" comment to be more
useful and more accurate: we've implemented the "nontrivial complexity"
of purging site-packages, and the other piece about nothing being
`pip install`-ed feels self-explanatory enough.
Differential Revision: https://phabricator.services.mozilla.com/D140580
So far, we've been using `virtualenv`'s `activate_this.py` script.
However, unlike earlier expectations, it adds its `sys.path` additions
to the //front//, not the back! This breaks our prioritization
requirements, such as:
* When using any package from the system environment, import *all
possible* packages from the system to avoid compatibility issues.
* Use vendored packages instead of virtualenv-installed packages
wherever possible, because it more-closely matches developer
expectations ("why is this package vendored if it's not used?")
Define an `activate_virtualenv()` function that replicates the logic
of `activate_this.py` [1], except for three differences:
* Don't modify `sys.real_prefix`, since it's a non-standard property of
`sys`.
* Only add seen-with-`venv`-module paths to the `sys.path` (`$prefix`,
`$prefix/.../$site_packages_dir`) - don't do the paths in-between.
* And, of course, append instead of prepend `sys.path` entries.
As an aside, this is one of the few remaining blockers from allowing
us to fully embrace `venv` instead of `virtualenv` - the last piece is
waiting on the fix for bug 1697833 to propagate.
[1]
https://github.com/pypa/virtualenv/blob/20.7.2/src/virtualenv/activation/python/activate_this.py
Differential Revision: https://phabricator.services.mozilla.com/D140579
Unless either the site's requirements or its on-disk virtualenv
changes, it shouldn't become out-of-date.
This will also catch situations where sites are incorrectly
being marked out-of-date due to `sys.path` mismanagement.
Differential Revision: https://phabricator.services.mozilla.com/D140578
When calling Mach, it was prefixed with the `srcdir`, even though the
`srcdir` was also the working directory of the invocation. This caused
the invocation to fail.
Differential Revision: https://phabricator.services.mozilla.com/D140400
Inconsistency confuses some of our tools. As part of this, I:
* Updated some docs to point to rust-minidump
* Added a fallback to mozcrash.py to try both versions
* Make mozcrash.py use --brief output when the local mdsw is used
* Remove the renaming hack from build-minidump-stackwalk.sh
This isn't as simple as a sed because we still have breakpad in tree
for minidump-analyzer. I did my best to replace the right strings.
Differential Revision: https://phabricator.services.mozilla.com/D138971
Updates parser documentation to clarify that, if no `paths` are
provided, then only those discovered by `--outgoing` and `--workdir`
are linted.
Differential Revision: https://phabricator.services.mozilla.com/D139158
Tie into `code_analysis`'s `get_clang_tools()` functionality to
intelligently bootstrap clang if it either doesn't exist or is
out-of-date.
This required exposing `command_context` to the linting logic, as it's
needed to call `artifact_toolchain(...)`.
Note that this means that the standalone `runcli.py` file won't be able
to support bootstrapping `clang-format`, or other linters that lean on
`command_context` in the future.
Finally, `substs.get("HOST_BIN_SUFFIX")` was replaced with a
windows-specific `binary += ".exe"`, because not all contexts where
the tests are run will have access to populated `substs` data.
Note that this worked before without the extension because it was
only used for starting a process, in which context Windows automatically
tries all `PATHEXT` options. Since we're now doing an `isfile()` check
(to enable more intelligent failure cases when `clang-format` doesn't
exist), we need the path to be fully correct.
Differential Revision: https://phabricator.services.mozilla.com/D137335
This unifies the ZIP permissions in omnijars and internal XPIs
produced by the packager directly and those repacked by l10n
single-repacks.
Differential Revision: https://phabricator.services.mozilla.com/D137340
Before, clang tools would only be bootstrapped if they didn't exist.
Now, bootstrapping also occurs if the version doesn't meet requirements.
Differential Revision: https://phabricator.services.mozilla.com/D137331
* Restructure "Using third-party Python packages" page to focus on the
"Mach commands"/"adding a Python package" use case since that's why
most people will be looking at these docs.
* Document the `<site>_virtualenv_packages.txt` behaviour and how it
relates to a Mach command's definition.
* Simplify the information around using a non-PyPI index to reference
the RelEng docs directly. It's a shame that the existing docs don't
explain how to identify tasks that need to use the internal mirror,
because I'm not sure either. There's existing cases of ad-hoc `pip`
installs //not// using the mirror, but the pattern isn't clear to me.
* Remove the "specify hashes" information, since the centralized
solution (will) automatically manage this internally.
* Arguably, it's still beneficial instructions for ad-hoc
`pip install` usages, but those are frowned upon today anyways - use
the centralized solution!
Differential Revision: https://phabricator.services.mozilla.com/D138931
This change allows IPDL sources to respect FINAL_LIBRARY when building, which
is important for allowing us to build gtest-only IPDL_SOURCES files.
Differential Revision: https://phabricator.services.mozilla.com/D137167
This patch adds support for generated files to be placed in
UNIFIED_SOURCES. This will be used in part 3 to build .cpp files
generated by IPDL_SOURCES by adding them to IPDL's UNIFIED_SOURCES under
the hood. Using a unified build for IPDL files is important, as a
significant build time regression was observed locally when using
non-unified sources to build ipdl sources, due to the large number of
generated files.
Differential Revision: https://phabricator.services.mozilla.com/D137166
This replaces all of the functions in gecko_taskgraph/util/taskcluster.py with
the ones from the vendored taskgraph if they are identical.
Differential Revision: https://phabricator.services.mozilla.com/D138458
This replaces all of the functions in gecko_taskgraph/util/taskcluster.py with
the ones from the vendored taskgraph if they are identical.
Differential Revision: https://phabricator.services.mozilla.com/D138458
`coverage` has native code, so the vendored version was only used as
source code from which the actual package could be built.
Since its always used in a context where we can `pip install` over the
network, let's do that. This cleans up our tree a bit and allows us to
leverage the `coverage` wheels.
Differential Revision: https://phabricator.services.mozilla.com/D138816
Following the pytest "deprecations and removals" docs [1], this patch:
* Replaces `@pytest.yield_fixture` with `@pytest.fixture`.
* Replaces `.funcargnames` with `.fixturenames`.
* Uses `Pathlib` parameter instead of the `py.path.local` one in
associated `pytest_*` hooks.
* Replaces `--strict` with `--strict-markers`
[1] https://docs.pytest.org/en/latest/deprecations.html
Differential Revision: https://phabricator.services.mozilla.com/D138815
If previous config file paths were generated with a different
driveletter casing than the current context, then each file's contents
are replaced with `null`.
MozillaBuild 3.4 used a lowercase drive letter, and MozillaBuild 4.0
will use an uppercase drive letter.
So, using 3.4's existing objdir, and doing an incremental build with
4.0 will cause the `null` contents issue.
Adjust the files to be their `Path().resolve()`'d counterparts, so
they're string-comparable and the subset-calculation works correctly.
Differential Revision: https://phabricator.services.mozilla.com/D137807
* The python2/python3 lookup logic is obsolete, we can always depend on
`sys.executable` since Mach starts with a Python version check
* `MozbuildObject`'s `python3` property was unused, which means that all
of this removed code was dead anyways.
Differential Revision: https://phabricator.services.mozilla.com/D138824
It seems sfv 0.8.0 is used in the tree, while neqo is using the newer 0.9.1.
Updating http-sfv to use sfv 0.9.1 should remove the code duplication.
Differential Revision: https://phabricator.services.mozilla.com/D139027
This will give additional feedback to the user. Now they will have
immediate feedback in regards to the progress of the test run, instead
of being unsure of what's happening until the first test run completes.
Differential Revision: https://phabricator.services.mozilla.com/D137925
Since stdout is being piped, redirecting stderr to stdout causes it
to end up in the same pipe interleaved, which is what we want.
Also added a non-zero return code to be set when there is no test output.
Differential Revision: https://phabricator.services.mozilla.com/D138481
