Bug 1713173 - Add --profile-command flag for profiling mach commands. r=firefox-build-system-reviewers,mhentges

I chose to do this at the level of the outer Python invocation because:

1. `python -m cProfile ...` handles writing the file and some other
   details.  It's possible to rebuild the functionality -- the tools
   are there -- but the APIs are awkward.

2. this allows to profile `mach` internals, instead of just the
   invoked command's implementation.

This uses the return code of the `get_command` subshell to transmit
the single bit of information "is the flag present".

The Python-level argument is required in order to have `--help` know
about the option and to avoid the `mach` shell script having to filter
arguments.

Differential Revision: https://phabricator.services.mozilla.com/D116151

mach

@@ -25,7 +25,7 @@ run_py() {
     shift
     if command -v "$py_executable" > /dev/null
     then
-        exec "$py_executable" "$0" "$@"
+        exec "$py_executable" $py_profile_command_args "$0" "$@"
     else
         echo "This mach command requires $py_executable, which wasn't found on the system!"
         case "$py_executable" in
@@ -60,6 +60,9 @@ get_command() {
         --log-no-times) shift;;
         -h) shift;;
         --debug-command) shift;;
+        --profile-command)
+            py_profile_command="1"
+            shift;;
         --settings)
             if [ "$#" -lt 2 ]
             then
@@ -73,10 +76,25 @@ get_command() {
         *) echo $1; break;;
     esac
     done
+    return ${py_profile_command}
 }
 
 state_dir=${MOZBUILD_STATE_PATH:-~/.mozbuild}
 command=$(get_command "$@")
+py_profile_command=$?
+
+if [ ${py_profile_command} -eq 0 ]
+then
+    py_profile_command_args=""
+else
+    # We would prefer to use an array variable here, but we're limited to POSIX.
+    # None of our arguments have quoting or spaces so we can safely interpolate
+    # a string instead.
+    py_profile_command_args="-m cProfile -o mach_profile_${command}.cProfile"
+    echo "Running with --profile-command.  To visualize, use snakeviz:"
+    echo "$HOME/.mozbuild/_virtualenvs/mach/bin/python -m pip install snakeviz"
+    echo "$HOME/.mozbuild/_virtualenvs/mach/bin/python -m snakeviz mach_profile_${command}.cProfile"
+fi
 
 # If MACH_USE_SYSTEM_PYTHON or MOZ_AUTOMATION are set, always use the
 # python 3 executables and not the virtualenv locations.

python/mach/docs/faq.rst

@@ -20,6 +20,31 @@ Core <https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Build%20Syste
    General*. Bugs against testing commands should be filed somewhere in
    the *Testing* product.
 
+How do I debug a command failing with a Python exception?
+---------------------------------------------------------
+
+You can run a command and break into ``pdb``, the Python debugger,
+when the command is invoked with:
+
+.. code-block:: shell
+
+   ./mach --debug-command FAILING-COMMAND ARGS ...
+
+How do I profile a slow command?
+--------------------------------
+
+You can run a command and capture a profile as the ``mach`` process
+loads and invokes the command with:
+
+.. code-block:: shell
+
+   ./mach --profile-command SLOW-COMMAND ARGS ...
+
+Look for a ``mach_profile_SLOW-COMMAND.cProfile`` file.  You can
+visualize using `snakeviz <https://jiffyclub.github.io/snakeviz/>`__.
+Instructions on how to install and use ``snakeviz`` are printed to the
+console, since it can be tricky to target the correct Python virtual
+environment.
 
 Is ``mach`` a build system?
 ---------------------------

python/mach/mach/main.py

@@ -686,6 +686,11 @@ To see more help for a specific command, run:
             action="store_true",
             help="Start a Python debugger when command is dispatched.",
         )
+        global_group.add_argument(
+            "--profile-command",
+            action="store_true",
+            help="Capture a Python profile of the mach process as command is dispatched.",
+        )
         global_group.add_argument(
             "--settings",
             dest="settings_file",