CCACHE_CPP2 is no longer a thing and hasn't been for ~10y.
There are other ways to configure icecream that isn't CCACHE_PREFIX, so
remove the broken and special-casey detection while at it.
The right thing to check would be to run `ccache -k prefix_command` or
so (and even so `mk_add_option` would bypass it).
It seems nobody had regenerated the metrics doc in a while...
Differential Revision: https://phabricator.services.mozilla.com/D295254
Added one field:
- The `--target` of a build.
Re-added `python/mach/metrics.yaml` to `./mach doc mach-telemetry` generation (Not sure why it was missing).
Re-generated the associated docs. (It looks like this was not done for previously when other fields were added.
Differential Revision: https://phabricator.services.mozilla.com/D208811
This is really just shuffling a bunch of things around. None of the
'load_*' member functions of the `Mach` class actually needed to be
member functions. They can all be static so that they can be used
anywhere. That combined with moving all the other 'mach_command' logic
to a different file, allows us to load the module for any command so
that we can successfully dispatch it.
Differential Revision: https://phabricator.services.mozilla.com/D184060
* Restructure "Using third-party Python packages" page to focus on the
"Mach commands"/"adding a Python package" use case since that's why
most people will be looking at these docs.
* Document the `<site>_virtualenv_packages.txt` behaviour and how it
relates to a Mach command's definition.
* Simplify the information around using a non-PyPI index to reference
the RelEng docs directly. It's a shame that the existing docs don't
explain how to identify tasks that need to use the internal mirror,
because I'm not sure either. There's existing cases of ad-hoc `pip`
installs //not// using the mirror, but the pattern isn't clear to me.
* Remove the "specify hashes" information, since the centralized
solution (will) automatically manage this internally.
* Arguably, it's still beneficial instructions for ad-hoc
`pip install` usages, but those are frowned upon today anyways - use
the centralized solution!
Differential Revision: https://phabricator.services.mozilla.com/D138931
When installing from Python.org, the `python` command isn't available by
default (in favour of `py`). Support using `py` instead.
There were issues running `hg` when `python` isn't added to the `PATH`,
so the docs are being updated accordingly.
Of course, it's still worth supporting `py`-instead-of-`python`
workflows for `git` users who won't be impacted by that bug.
Differential Revision: https://phabricator.services.mozilla.com/D136286
Assuming that the `MOZILLABUILD` environment variable is set, allow
invoking Mach from non-MozillaBuild terminals.
Note that MozillaBuild still needs to be installed, and the
`MOZILLABUILD` environment variable will have to be set.
For future reference: when I tried setting this up with Windows
Store's Python 3.9, I encountered issues when running binaries installed
via `pip`: it would fail with `abort: failed to load Python DLL
python3x.dll`.
Differential Revision: https://phabricator.services.mozilla.com/D133936
As part of this, the shell-script part of `./mach` can be removed,
making it pure Python.
There's a change in `--profile-command` behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because _so much of Mach_ has already been run before
CLI arguments are parsed in the Python process.
If a developer wants to profile Mach itself, they can manually run
`python3 -m cProfile -o <file> ./mach ...`
Differential Revision: https://phabricator.services.mozilla.com/D133928
As part of this, the shell-script part of `./mach` can be removed,
making it pure Python.
There's a change in `--profile-command` behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because _so much of Mach_ has already been run before
CLI arguments are parsed in the Python process.
If a developer wants to profile Mach itself, they can manually run
`python3 -m cProfile -o <file> ./mach ...`
Differential Revision: https://phabricator.services.mozilla.com/D133928
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
I chose to do this at the level of the outer Python invocation because:
1. `python -m cProfile ...` handles writing the file and some other
details. It's possible to rebuild the functionality -- the tools
are there -- but the APIs are awkward.
2. this allows to profile `mach` internals, instead of just the
invoked command's implementation.
This uses the return code of the `get_command` subshell to transmit
the single bit of information "is the flag present".
The Python-level argument is required in order to have `--help` know
about the option and to avoid the `mach` shell script having to filter
arguments.
Differential Revision: https://phabricator.services.mozilla.com/D116151
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
`mach doc mach-telemetry` wasn't run the last time the telemetry
changed. By running it now, the docs are now up-to-date.
Differential Revision: https://phabricator.services.mozilla.com/D106544
This document was imported from MDN and contained very outdated/incorrect information, and much of the information here is duplicated from the existing `mach` documentation. For the little content that isn't already expressed in the existing documentation in a better way, merge it into `python/mach/docs`.
The unique content is mainly in the FAQ, so I added a new page for that.
Differential Revision: https://phabricator.services.mozilla.com/D91455
Allows mach commands to define their own glean metrics with the `metrics_path` @CommandProvider parameter.
When `metrics_path` is defined:
* A `metrics` kwarg is provided to the decorated class. This `metrics` handle is a Glean instance, so Glean documentation should be consulted for usage information.
* When `mach doc telemetry` is run, metrics docs will be generated from all the registered metrics files.
Note: there was some consideration between making `metrics_path` a @CommandProvider or @Command parameter.
In the end, @CommandProvider seemed like a better fit because:
* Metrics seem to be more associated with the entire class than a specific command/method. This is because a class represents a "domain", and that domain may have different commands that have overlapping metrics. Accordingly, all the metrics should be defined once as available to the entire class.
* Currently, @Command methods only take parameters that map one-to-one with CLI arguments. It could seem inconsistent to have one exception: the metrics handle
Differential Revision: https://phabricator.services.mozilla.com/D85953
In addition to the existing build telemetry, also gather the stats and
report with Glean. This new telemetry is reported in tandem with the existing
telemetry to allow testing and confidence before a full roll-out.
Additionally, Glean isn't compatible with Python 2, so the new telemetry only runs
on Python 3 mach commands.
Differential Revision: https://phabricator.services.mozilla.com/D83572
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.
Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.
Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.
Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D86255
Alex Hochheiden