This patch adds support for generated files to be placed in
UNIFIED_SOURCES. This will be used in part 3 to build .cpp files
generated by IPDL_SOURCES by adding them to IPDL's UNIFIED_SOURCES under
the hood. Using a unified build for IPDL files is important, as a
significant build time regression was observed locally when using
non-unified sources to build ipdl sources, due to the large number of
generated files.
Differential Revision: https://phabricator.services.mozilla.com/D137166
This replaces all of the functions in gecko_taskgraph/util/taskcluster.py with
the ones from the vendored taskgraph if they are identical.
Differential Revision: https://phabricator.services.mozilla.com/D138458
This replaces all of the functions in gecko_taskgraph/util/taskcluster.py with
the ones from the vendored taskgraph if they are identical.
Differential Revision: https://phabricator.services.mozilla.com/D138458
`coverage` has native code, so the vendored version was only used as
source code from which the actual package could be built.
Since its always used in a context where we can `pip install` over the
network, let's do that. This cleans up our tree a bit and allows us to
leverage the `coverage` wheels.
Differential Revision: https://phabricator.services.mozilla.com/D138816
Following the pytest "deprecations and removals" docs [1], this patch:
* Replaces `@pytest.yield_fixture` with `@pytest.fixture`.
* Replaces `.funcargnames` with `.fixturenames`.
* Uses `Pathlib` parameter instead of the `py.path.local` one in
associated `pytest_*` hooks.
* Replaces `--strict` with `--strict-markers`
[1] https://docs.pytest.org/en/latest/deprecations.html
Differential Revision: https://phabricator.services.mozilla.com/D138815
If previous config file paths were generated with a different
driveletter casing than the current context, then each file's contents
are replaced with `null`.
MozillaBuild 3.4 used a lowercase drive letter, and MozillaBuild 4.0
will use an uppercase drive letter.
So, using 3.4's existing objdir, and doing an incremental build with
4.0 will cause the `null` contents issue.
Adjust the files to be their `Path().resolve()`'d counterparts, so
they're string-comparable and the subset-calculation works correctly.
Differential Revision: https://phabricator.services.mozilla.com/D137807
* The python2/python3 lookup logic is obsolete, we can always depend on
`sys.executable` since Mach starts with a Python version check
* `MozbuildObject`'s `python3` property was unused, which means that all
of this removed code was dead anyways.
Differential Revision: https://phabricator.services.mozilla.com/D138824
It seems sfv 0.8.0 is used in the tree, while neqo is using the newer 0.9.1.
Updating http-sfv to use sfv 0.9.1 should remove the code duplication.
Differential Revision: https://phabricator.services.mozilla.com/D139027
This will give additional feedback to the user. Now they will have
immediate feedback in regards to the progress of the test run, instead
of being unsure of what's happening until the first test run completes.
Differential Revision: https://phabricator.services.mozilla.com/D137925
Since stdout is being piped, redirecting stderr to stdout causes it
to end up in the same pipe interleaved, which is what we want.
Also added a non-zero return code to be set when there is no test output.
Differential Revision: https://phabricator.services.mozilla.com/D138481
Logs from CI tasks have each line prefixed with date and time. Tasks like
Python linting log another timestamp how long the script has been running. This
should be omitted in automation to align it with log formats of other tasks
like mochitests which integrates it better with expectations of Treeherder's
log parser and bug filer..
Differential Revision: https://phabricator.services.mozilla.com/D138444
If there's no files to lint in CI, that's a smell that something is
misconfigured.
Resolve this by raising an exception accordingly if `MOZ_AUTOMATION` is
set (and we aren't running the tests for the linting code itself).
Automatically recommend specifying `*`, as (assuming the use of a
shell) that will resolve to "all files/dirs at $topsrcdir`", which
matches existing behaviour if `--outgoing` and `--workdir` aren't
provided.
Differential Revision: https://phabricator.services.mozilla.com/D138172
This patch adds the --proxy-perftest-page option. With this option, we'll be able to specify which test pages we want to record rather than modifying the pageload_sites.json. Note that the login fields will not be taken into consideration with this option.
You can use the flag as follows (seperate multiple pages with a comma): --proxy-perftest-page microsft,linkedin,netflix
Furthermore, some changes were made to strengthen the conditions around when a login site can be tested (only when RAPTOR_LOGINS is defined locally, or if we are in CI).
Differential Revision: https://phabricator.services.mozilla.com/D137468
There's a bug with `ProcessHandler` on Windows that sometimes causes
tests to hang for an additional 3 minutes. This is a workaround that
just replaces `ProcessHandler` with a standard Python3 approach that
achieves the same thing.
Also added some logging output as the threads are created as some
additional feedback. Prior to this, the tests silently started
running without any indication.
Differential Revision: https://phabricator.services.mozilla.com/D137822
The new `mozlint` behaviour will always lean on the VCS
to only lint changed files.
However, in tests, the VCS is going to be slow and also provide
inconsistent data between tests. Additionally, in some contexts,
the VCS won't be available.
Resolve this by hardcoding a file to lint for each case.
Differential Revision: https://phabricator.services.mozilla.com/D138033
Python 3.10 prints a warning when it sees `distutils` used, so let's
avoid using `distutils` in the "common Mach code" that runs on every
Mach call.
Note that this change will also fix the `SetuptoolsDeprecationWarning`
that shipped on `setuptools>=58.3.0`, since our usage of `distutils` was
calling a `setup.py install` under the hood.
Differential Revision: https://phabricator.services.mozilla.com/D137499
Tweaks `./mach lint` behaviour to always fall back to only linting files
that have changed according to VCS - previously, this only happened if
no linter was provided.
Adjusts "am I at $topsrcdir" check to use `pathlib` to avoid mismatches
due to inconsistent capitalization or slash direction.
Updates CI references to explicitly provide `*` as the path to avoid
the only-lint-files-changed restriction.
Differential Revision: https://phabricator.services.mozilla.com/D137870
The main difference between `OSXBootstrapperLight` and `OSXBootstrapper`
is that the "light" one doesn't install utilities via `brew`.
However, these utilities are all optional for builds (`watchman`,
`terminal-notifier`, etc).
Since arm macOS + Rosetta + brew installs are failing (because brew is
complaining that the "ARM prefix" is being used in a Rosetta context),
the solution provided by this patch is to avoid using `brew` in this
case.
Differential Revision: https://phabricator.services.mozilla.com/D137661
In addition to showing the `pip check` failure, also show:
* The `pip install --requirements ...` output
* The output of `pip list -v`
* The path to the `requirements.txt` file and the affected site.
To get sufficient detail of `pip install --requirements` to display
if the subsequent `pip check` call failed, I had to remove the
`--quiet` parameter. It was replicated by hiding _all_ output
if `quiet=True` and `pip install` succeeds - this is different to
the previous behaviour, where warnings would still poke through,
even if `quiet=True` and the install succeeded.
This was the only way to make it work such that all output would
be available without having to run `pip install` twice (which would
likely get different output the second time). Besides, there weren't
many warnings triggering this edge case, so the change in behaviour is
probably acceptable.
Additionally, in both places doing `pip check`
(post-install-requirements and when doing system-package compatibility
checking), move the `pip check` output to happen last, so that users
don't need to scroll to the top of the `pip list` wall of text to find
the reason for their command failure.
Differential Revision: https://phabricator.services.mozilla.com/D137206
This reduces some code duplication and paves the way for modifying how
Mach loads modules.
The plan is to follow this up by making this function take a more fleshed out 'spec'
that maps command names to the files they're defined at and possibly more metadata.
Since this may affect how Mach works internally (e.g. with lazy loading in the roadmap),
it makes sense to move this logic inside the Mach class.
Differential Revision: https://phabricator.services.mozilla.com/D136789
Though not a panacea, this patch addresses existing usages of
`get_repository_from_build_config()` to fall back to
`get_repository_object()` if `MissingVCSTool` was encountered.
Ideally, we'd update `substs["HG"]` when we detect that it's changed,
but we don't have infrastructure to incrementally change configure
output.
So, other issues may creep up that will still warrant a fresh
`./mach configure`.
Differential Revision: https://phabricator.services.mozilla.com/D137675
Implements keep key in moz.yaml for keeping pre-existing files in the vendor directory
Implements include key in moz.yaml for forcing inclusion of files and folders from upstream
Differential Revision: https://phabricator.services.mozilla.com/D124799
The py.path allowed adding arbitrary fields to it, while a pathlib.Path
does not. To replicate the behavior with as few changes as possible, I
made a test fixture wrapper class (RepoTestFixture) to add the same
arbitrary fields to, while now containing a Pathlib path instead.
Also added a utility functions to make the running of tests a
bit cleaner ('execute_next_step')
Differential Revision: https://phabricator.services.mozilla.com/D136302
Note: In `test_commit.py` For date formatting, the `%s` did not work on Windows.
On Linux it gives the time in seconds. The switch to the rfc822date format for
both hg and git `%aD` made the expected output in the check later work for both
git and hg, and made it platform agnostic.
Differential Revision: https://phabricator.services.mozilla.com/D136209
The compression options appear to be the same: `mozjar` defaults to
level 9 compression. The `mozjar` code is definitely slower than the
native ZIP executable, but not, I think, by enough to not do this.
This is the sole consumer of `make_zip.py` in the tree, so we remove
the file entirely.
Differential Revision: https://phabricator.services.mozilla.com/D118528
This keeps the shape of the package produced by `mach package` in the
artifact build like that of the input `Bs` package.
Differential Revision: https://phabricator.services.mozilla.com/D136671
We support splitting `MOZ_ARTIFACT_FILE` by `os.pathsep` and splitting
`MOZ_ARTIFACT_URL` by space (`" "`) to allow both the package and the
tests.common archives to be specified.
Sadly, `file://` URLs are not (yet) accepted, so there's need for the
sibling environment variables.
Differential Revision: https://phabricator.services.mozilla.com/D135939
Currently, developers don't have a way to have the Mach virtualenv
re-attempt to install optional dependencies (such as `glean`).
As part of `./mach bootstrap` (the general catch-all "re-create my dev
environment" command), we should retry installing optional dependencies.
This also matches the "glean isn't installed" error message
recommendation.
Note: This doesn't address the case in which command virtualenvs
need their optional dependencies attempted to be reinstalled.
However, since we don't have any such cases yet, I'm satisfied with
deferring that work.
Differential Revision: https://phabricator.services.mozilla.com/D123242
Our platform-specific `bootstrap` code transitively depends on
the standard library's `distutils` module, but Debian Linux and derived
distros generally split `distutils` into a separate installable OS
package from `python3` itself.
So, Python 3 being able to run isn't a sufficient guarantee that
`distutils` is available.
To catch this case, add a `distutils` check at the very beginning of
Mach initialization.
-----
Also remove an obsolete comment claiming that `mach bootstrap` doesn't
need `distutils`, which was obsoleted by Bug 1717051.
Differential Revision: https://phabricator.services.mozilla.com/D136639
What this function does can easily be a one-liner now that we're using
Pathlib, so both calls to it have been replaced by said one-liner.
Differential Revision: https://phabricator.services.mozilla.com/D135263
When installing from Python.org, the `python` command isn't available by
default (in favour of `py`). Support using `py` instead.
There were issues running `hg` when `python` isn't added to the `PATH`,
so the docs are being updated accordingly.
Of course, it's still worth supporting `py`-instead-of-`python`
workflows for `git` users who won't be impacted by that bug.
Differential Revision: https://phabricator.services.mozilla.com/D136286
The main thread is blocking (for an unknown reason) without piping the entirety of the log
file to `less` (unless the user manually scrolls to almost the bottom). Since the piping
cannot always complete without a user terminating, we must hide the errors that arise
from using the `stdin` stream for the `less` process after the process is terminated.
Differential Revision: https://phabricator.services.mozilla.com/D135979
- Changed byte strings being set in env to just strings
as byte strings were only necessaery in Python2 and
strings work fine in POSIX and Windows as of Python3.
(Byte strings no longer work on Windows in Python3)
- Replaced all the file descriptor logic used to pipe
to `less` with more Pythonic stream manipulation.
(There was a change in Windows console IO in Python 3.6
that broke the file descriptor logic we were using here)
See: https://www.python.org/dev/peps/pep-0528/#add-io-windowsconsoleio
Differential Revision: https://phabricator.services.mozilla.com/D135969
Sphinx already runs within the `./mach doc` command, which has the
`docs` virtualenv activated. Activating another virtualenv within a
Sphinx python file is redundant.
Differential Revision: https://phabricator.services.mozilla.com/D135305
When the redundant-site-package-dir pruning was originally implemented,
it assumed that only `<venv>/lib/<site-packages-dir>` was automatically
added to the `sys.path`. However, it looks like some platforms
(including Windows) also add the prefix of the venv itself to the
`sys.path`.
We can work around this by removing both from the `sys.path` carry-over,
and the deprioritization logic.
Differential Revision: https://phabricator.services.mozilla.com/D135300
The sym files generated for Thunderbird builds contain incorrect VCS references.
symbolstore.py will handle the subrepo correctly with an additional -s argument
if MOZ_SOURCE_REPO and MOZ_SOURCE_CHANGESET are not set. See D135392.
Differential Revision: https://phabricator.services.mozilla.com/D135299
Currently, the bootstrap code from configure gets bootstrapped task info
from the taskgraph, and then invokes `mach artifact toolchain` which...
does the same again. That cumulatively wastes a noticeable amount of
time, especially on Windows.
So to avoid repeating this work that the configure side already did,
we do a little more on the configure side (resolution from index to
task-id), and just give a (task-id, artifact path) pair to `mach
artifact toolchain`. The added code to `mach artifact toolchain` is
actually ironically very similar to the code that was removed in
bug 1687594.
And now that mach bootstrap uses the configure code, it benefits from
this change as well.
Differential Revision: https://phabricator.services.mozilla.com/D135311
The python-test infra runs each test file in a separate process for
parallelization. This is expedient, but it means that when a test fails, the
output is often buried 1000s of lines in scrollback and difficult to find.
In the future, we could look into using 'pytest-async' rather than rolling our
own parallelization, this way Pytest would output the failures again at the end
of the run.
But for now, it's easier to simply defer outputting failed tests until the very
end. This is done in a try/finally block so even if there was a
KeyboardInterrupt or other exception, we'll still print the failed tests.
Differential Revision: https://phabricator.services.mozilla.com/D135274
MSYS2 has a slightly different directory structure (binaries are
now under "/usr/bin/" instead of "/bin/"), and we're now plonking
it in `%MOZILLABUILD%\msys2` instead of `%MOZILLABUILD%\msys` so that
MSYS1 files don't interfere with MSYS2 after a pave-over install.
Speaking of pave-over installs: if both `msys2` and `msys` are available,
this patch prefers `msys2`. This is because MozillaBuild installations
with MSYS2 are going to _assume_ they're using MSYS2, and therefore
be most compatible with the versions of packages shipped with MSYS2.
Differential Revision: https://phabricator.services.mozilla.com/D133549
Assuming that the `MOZILLABUILD` environment variable is set, allow
invoking Mach from non-MozillaBuild terminals.
Note that MozillaBuild still needs to be installed, and the
`MOZILLABUILD` environment variable will have to be set.
For future reference: when I tried setting this up with Windows
Store's Python 3.9, I encountered issues when running binaries installed
via `pip`: it would fail with `abort: failed to load Python DLL
python3x.dll`.
Differential Revision: https://phabricator.services.mozilla.com/D133936
As part of this, the shell-script part of `./mach` can be removed,
making it pure Python.
There's a change in `--profile-command` behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because _so much of Mach_ has already been run before
CLI arguments are parsed in the Python process.
If a developer wants to profile Mach itself, they can manually run
`python3 -m cProfile -o <file> ./mach ...`
Differential Revision: https://phabricator.services.mozilla.com/D133928
Currently, using any of the functions defined in
`ConfigureSandbox.OS.path` in a `@depends` function will have a different
behavior if the function has an `@imports("os")` or not. In the former
case, we get plain `os.path.$function`, while in the latter we get the
function from `ConfigureSandbox.OS.path`, which handles path separators
differently, which makes a significant difference on Windows.
With this change, we now consistently use the versions from
`ConfigureSandbox.OS.path`.
Differential Revision: https://phabricator.services.mozilla.com/D135003
The VS backend is always built when building on Windows for Windows, so
in practice, the message isn't printed... except when cross-compiling on
Windows, in which case the VS backend doesn't work.
Differential Revision: https://phabricator.services.mozilla.com/D135015
Currently, using any of the functions defined in
`ConfigureSandbox.OS.path` in a `@depends` function will have a different
behavior if the function has an `@imports("os")` or not. In the former
case, we get plain `os.path.$function`, while in the latter we get the
function from `ConfigureSandbox.OS.path`, which handles path separators
differently, which makes a significant difference on Windows.
With this change, we now consistently use the versions from
`ConfigureSandbox.OS.path`.
Differential Revision: https://phabricator.services.mozilla.com/D135003
pkgconf is an alternative implementation of pkg-config that is more
cross-platform. It has also become the default on Fedora, so it's not
some random project.
Differential Revision: https://phabricator.services.mozilla.com/D135009
pkgconf is an alternative implementation of pkg-config that is more
cross-platform. It has also become the default on Fedora, so it's not
some random project.
Differential Revision: https://phabricator.services.mozilla.com/D135009
As part of this, the shell-script part of `./mach` can be removed,
making it pure Python.
There's a change in `--profile-command` behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because _so much of Mach_ has already been run before
CLI arguments are parsed in the Python process.
If a developer wants to profile Mach itself, they can manually run
`python3 -m cProfile -o <file> ./mach ...`
Differential Revision: https://phabricator.services.mozilla.com/D133928
It occurs to me that because tracking is a general vendoring need
(not just for updatebot) that it should live under vendoring.
Depends on D129437
Differential Revision: https://phabricator.services.mozilla.com/D129533
- Avoid the flag selection silently not enabling the new pass manager
when --enable-new-pass-manager is passed explicitly.
- Avoid adding the -fexperimental-new-pass-manager to clang >= 13, which
has it enabled by default. Likewise for the linker flags.
- Remove the force-enable of the new pass manager with clang < 12 on
automation, since we're using version 13 anyways.
- Account for the fact that both lld and ld64 can pass the
-import-hot-multiplier flag to the LTO plugin on mac builds, which
effectively will set it for the first time on mac, and might improve
performance.
Differential Revision: https://phabricator.services.mozilla.com/D134860
`pip install`'s standard behaviour is to satisfy the new requirement,
even if it conflicts with existing packages. If a conflict _does_
occur, `pip` will simply warn about it after the installation has
already damaged the environment.
By using the `--constraint` feature, we force `pip` to consider
already-installed packages before installing new ones. If a conflict
is detected, an error is raised and `pip` stops without damaging
the environment.
Since we're capturing system packages (when needed) with this feature,
we can now allow virtualenvs that do pip-installations to safely work
with system-using Mach environments.
Differential Revision: https://phabricator.services.mozilla.com/D126925
Port some `sys.path` modifications to the centralized system.
Not only is this cleaner (ad-hoc global modifications are spicy), but
this also avoids "reordering" issues that can creep in during nested
calls, causing virtualenvs to be unnecessarily recreated.
Differential Revision: https://phabricator.services.mozilla.com/D134481
NSS gyp files use -l$lib, and while OS_LIBS accepts this form and passes
it through, it's not actually a recognized way to link libraries on
clang-cl builds. So, re-normalize the values to not include -l, which
will add it back in the backend when appropriate, or switch to $lib.lib
on platforms that use this form.
Differential Revision: https://phabricator.services.mozilla.com/D134736
This patch resolves cases like the following:
1. The system Python has `zstandard`.
2. `MOZ_AUTOMATION=1 ./mach python --virtualenv psutil <script>`
is run, adding `psutil` to the import scope.
3. `<script>` runs Mach a _second_ time, and this time Mach needs to
be able to import `zstandard` (in this case, it should be able
to fetch it from the system Python's site-packages).
The previous behaviour would add the "site-packages" of the //invoking//
Python interpreter, but ancestor packages would get dropped.
To rectify this issue, this patch changes "import inheritance" to keep
more of the `sys.path`, rather than just
`<external-python>.all_site_packages_dirs()`.
Note: the original implementation of this patch passed forward *all*
of the `sys.path` when creating virtualenvs. However, this caused issues
when Mach ran `pip`, because `pip` was no longer able to discover the
"standard library" (it was failing because it assumed all paths defined
in a virtualenv's site were non-standard-library paths, and the original
implementation broke that assumption).
As part of this, a distinction was defined between the "current" Python
interpreter (external_python) and the top-level Python interpreter
(original_python). This was needed to enable discovering which paths
are "standard library paths".
Differential Revision: https://phabricator.services.mozilla.com/D134201
We clean up our Gecko clone between tasks via `hg robustcheckout --purge`, which runs `hg purge`. This is very effective, *but* it doesn't detect or clean up any nested clones.
Because we run cross-channel on Gecko workers, and because we clone `comm/` in cross-channel and haven't cleaned it up, and because `hg purge` doesn't detect or clean up nested clones, and because our current virtualenv setup code traverses the tree and can error out on `comm/` clones, let's clean up `comm/` after running cross-channel.
We'll be moving TB cross-channel to different tasks/workers in bug 1742711, and ideally we can update robustcheckout and/or `hg purge` to be able to detect and/or clean up nested clones.
Differential Revision: https://phabricator.services.mozilla.com/D134582
This uses the same trick as bug 1743832 under the hood. We could go the
other way around, extracting the configure code to a separate module,
but the longer term goal here is to have configure figure out which
things to bootstrap for the selected build type.
As a side effect, mach bootstrap will stop re-bootstrapping things that
are already up-to-date, at least for things using
install_toolchain_artifact, excluding those that don't follow the
convention wrt the extracted directory path.
Differential Revision: https://phabricator.services.mozilla.com/D134595
We clean up our Gecko clone between tasks via `hg robustcheckout --purge`, which runs `hg purge`. This is very effective, *but* it doesn't detect or clean up any nested clones.
Because we run cross-channel on Gecko workers, and because we clone `comm/` in cross-channel and haven't cleaned it up, and because `hg purge` doesn't detect or clean up nested clones, and because our current virtualenv setup code traverses the tree and can error out on `comm/` clones, let's clean up `comm/` after running cross-channel.
We'll be moving TB cross-channel to different tasks/workers in bug 1742711, and ideally we can update robustcheckout and/or `hg purge` to be able to detect and/or clean up nested clones.
Differential Revision: https://phabricator.services.mozilla.com/D134582
There are two sites that are allowed to define their dependencies
in a flexible way: the `mach` and `build` sites.
This is because these are the only two sites that _may_ have
to operate without `pip install`-ing any packages, and instead having
to be compatible with the packages installed to the system.
Due to this required compatibility, allowing flexibility to these sites
allows flexibility downstream.
Anyways, this patch isn't about that - that behaviour has already
landed. This patch is about tweaking `requirements.py` so that
*it* doesn't care about specific sites, but rather only cares about
if it should assert `only_strict_requirements` or not. Accordingly,
the helpful "not all packages are pinned" error message is moved
to `site.py`, where it belongs.
Differential Revision: https://phabricator.services.mozilla.com/D132082
Even if a command site has its own comfy virtualenv, if Mach is using
the system packages then they'll still be in-scope for commands.
So, we still need to `_assert_pip_check()` in case the command's
vendored packages conflict.
Differential Revision: https://phabricator.services.mozilla.com/D132168
Pip is able to detect unpacked sdists because they have a `.egg-info`
directory, *not* because they have a top-level `PKG-INFO` file.
This is confirmed by the `MarkupSafe` case, where pip can see the
package in `third_party/python/MarkupSafe/src`, even though there's no
`PKG-INFO` at that depth.
Make two other changes as part of this:
* Only submit the warning if the package is under a "third_party"
directory to avoid a false positive when the developer has performed
a "pip install -e" on a first-party module
* Move the check to `test_site_compatibility` to avoid unnecessary
validation at runtime.
Differential Revision: https://phabricator.services.mozilla.com/D126924
In CI, we sometimes don't have permissions to create a scoped state dir.
Additionally, the current behaviour for resolving the path to a scoped
state dir will also attempt to create it if it doesn't exist.
There's likely a more eloquent solution, but the short-term fix is to
have sites defer the resolution of the state dir until they _know_ they
need it.
Differential Revision: https://phabricator.services.mozilla.com/D134066
