We have a few bugs filed as intermittent failures all relating
to an `AssertionError` on the return value of this block, which
seems to have a relevant value when `OSError` is thrown. Removing
this will not fix the problem but will at least let us see the
true cause of the failure for debugging and stabilization.
Differential Revision: https://phabricator.services.mozilla.com/D11884
This change tries to ensure that we don't write telemetry data for mach
commands invoked recursively as part of other mach commands. The intent of
build system telemetry is to only collect data about commands that users are
invoking directly.
There are two ways that we found mach commands can be recursively invoked:
* By running a python subprocess to recursively invoke mach (used in
`mach bootstrap` to call `mach artifact toolchain`)
* By using `Registrar.dispatch` to delegate to a sub-command (used by many
build system commands to invoke `mach build`).
The subprocess case is handled here by having mach set a `MACH_MAIN_PID`
environment variable whose value is the current process' pid on startup if it
does not already exist in the environment. Telemetry code then checks that the
value of that variable matches the current pid and skips writing telemetry data
if not.
The dispatch case is handled by making `MachRegistrar` store the current depth
of the command stack and pass it to the `post_dispatch_handler` which will skip
writing telemetry data if depth != 1.
Additionally the `should_skip_dispatch` function in mach_bootstrap is renamed
to `should_skip_telemetry_submission`, which was its original intent. The
combination of checks added in this change should be sufficient for deciding
when to write telemetry data, and we were not collecting telemetry for the set
of mach commands in that function (which included `mach bootstrap`).
In order to facilitate writing a test for the dispatch case this change adds a
`mach python --exec-file` option to execute Python code directly in the context
of the `mach python` command.
Differential Revision: https://phabricator.services.mozilla.com/D11207
The build telemetry code attempts to filter paths to avoid PII from usernames
and other things. It does this by converting every commandline argument to
an absolute path and then making them relative to topsrcdir or topobjdir and
omitting any that fail. This meant that running a mach command with a cwd
outside of the topsrcdir or objdir would omit all arguments since they were
converted to absolute paths from the cwd.
This change fixes this by adding the cwd to the list of paths used to create
relative paths. Additionally we add the user's home directory to that list
to try to avoid usernames sneaking through. Finally, instead of simply
removing these path prefixes, we replace them with sigils: $topsrcdir,
$objdir, $HOME.
Differential Revision: https://phabricator.services.mozilla.com/D11174
This change adds python/mach/mach/test/test_telemetry.py which contains
a simple test that running mach with the build telemetry setting enabled
causes us to write a telemetry file. The test fixture in the file should
make it easy to write additional tests.
A necessary precursor to make the tests work was to change mach_bootstrap's
`should_skip_dispatch` function, which would refuse to write telemetry data
if stdout was not a terminal (which it isn't in the tests) and also in
automation. The latter test is moved to ensure that we don't *submit*
telemetry data from automation, but we can still write it to disk. Machines
in automation should never have the telemetry setting enabled outside of
these tests anyway, so this should not change anything in practice.
Differential Revision: https://phabricator.services.mozilla.com/D11172
Changes:
- under a specific code path such as:
`./mach <test_type> --debugger`
there exists now additional checks for IndexError when attempting to parse the debugger program name.
- print a nicer error message if user failed to properly specify the debugger program name.
- raise exit code of 1 if IndexError is raised.
Others:
- code comments updated to better reflect/describe what the section does.
Differential Revision: https://phabricator.services.mozilla.com/D10987
Behavior changes:
- instead of reading the mozconfig file, it now instantiates an instance of the build object.
- safe checking methods are used to access attributes to prevent errors on automation environment.
- better mach command parsing is performed with handler category instead of error-prone argv parsing.
Other changes:
- docstring for testing/xpcshell/runxpcshelltests.py::buildTestList() added and modernized.
- added clause that if length of tests gathered is 0, mach exits with an error code of 1.
Differential Revision: https://phabricator.services.mozilla.com/D7133
blessings.tigetstr is not part of its API. It happens to work because
blessings imports curses using 'from curses import tigetstr'.
Instead, we can just use terminal.normal, which contains the string we were
going to get anyway.
See https://github.com/erikrose/blessings/pull/138 for more information.
Let me know if there's a better way of resolving this. Hopefully with this +
the patch I submitted to blessings (https://github.com/erikrose/blessings/pull/137)
firefox will build fine with TERM improperly set.
Differential Revision: https://phabricator.services.mozilla.com/D5377
- added checkers in python/mach/mach/main.py prior to calling registrar.py.
- added internal function to check if specified debugger is installed.
- support both ./mach test <test_name> and ./mach <test_category> styles.
Differential Revision: https://phabricator.services.mozilla.com/D7234
- added checkers in python/mach/mach/main.py prior to calling registrar.py.
- added internal function to check if specified debugger is installed.
- support both ./mach test <test_name> and ./mach <test_category> styles.
Differential Revision: https://phabricator.services.mozilla.com/D7234
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
This patch allows executing |mach python-test| against Python 3 by specifying the optional |--three| command line option. When this option is present, pipenv will be used to manage a virtual environment using Python 3 and attempt to run the tests. When it is not present, pipenv will not be used, and everything will work as it did before this patch.
My original plan was to use pipenv regardless of the target version of Python, however I encountered several issues running some of the tests against our Python packages. Once all tests have been patched to run against Python 3, then we should be able to use pipenv when running them against Python 2.
Note that this patch allows tests to run against Python 3, but there are plenty of issues preventing them from passing. With this patch in place we can start to add Python 3 support to our packages and have the tests running in CI to ensure we don't regress back to just supporting Python 2.
MozReview-Commit-ID: BuU5gZK83hL
IHG: changed taskcluster/ci/source-test/python.yml
This is a new issue that gets linted with flake8 3.5.0. Basically you should
never use a blank except: statement.
This will catch all exceptions, including KeyboardInterrupt and SystemExit
(which is likely not intended). If a catch all is needed, use
`except: Exception`. If you *really* mean to also catch KeyboardInterrupt et
al, use `except: BaseException`.
Of course, being specific is often better than a catch all.
MozReview-Commit-ID: FKx80MLO4RN
Currently, marking a logger as a structured logger will require a
subsequent function call in order for the logger to be hooked up
to active handlers. This behavior is not intuitive and makes it
easy to not have handlers for newly-registered loggers. This means
messages may not be logged anywhere.
In addition, we have to manually specify which named loggers to
enable structured logging for. This can be annoying.
We change the behavior of register_structured_logger() to
automatically add existing terminal and json handlers to the
logger being marked as structured.
We also introduce an API to enable structured logging for all
loggers. Existing consumers of registered_structured_logger()
in mozbuild have been updated to use this API. A new consumer
has been added for the `mach configure` command because it should
have been there before.
We stop short of making enable_all_structured_loggers() the default.
This is because various commands interact with the log manager in
ways that will result in duplicate logging of messages and
dropping of structured messages. There is a bit of a rabbit hole
here and addressing it can be done as a follow-up.
MozReview-Commit-ID: 1aU6eJvTSMP
We don't use this attribute outside of this function. Besides, you can
easily obtain a handle on the logger by calling
``logging.getLogger('mach')``.
MozReview-Commit-ID: 41vdn6McowW
This passes `python3 -mcompileall`. Changes:
* use `0o` prefix for octal literals
* print as a function
* except .. as
* use six.reraise to replace a multi-argument raise statement
* use six.string_types and six.moves.configparser
* remove uses of `L` suffix for long integers
MozReview-Commit-ID: KLaYRNHGpay
This preserves ./mach settings' --list option. If --list is passed in, we call splitlines()
on the description and print only the first line. The full multi-line description will be
printed otherwise.
This also displays the type and/or choices of the option as appropriate.
MozReview-Commit-ID: 7UMsN9qslWt
We currently raise if we detect a command has both the `parser` attribute and a subcommand
at the same time, but as far as I can tell, there's no good reason to do this. Handling a
parser + subcommands seems to work exactly how you would expect. Furthermore, it isn't an
error to have subcommands + @CommandArgument, so it doesn't make sense that we're allowing
one but not the other.
This change solves an (admittedly unique) use case I'm trying to build into |mach try|. There
are N subcommands that all support a --save and --load style argument. So, e.g, we might have:
./mach try syntax --save foo
./mach try fuzzy --save bar
The main command will have the ability to detect which subcommand a saved value was generated
from and automatically dispatch to it. So this will work:
./mach try --load foo # dispatches to the syntax subcommand
./mach try --load bar # dispatches to the fuzzy subcommand
In order to share the --save/--load arguments across the main command + subcommands, we need
to set the parser attribute.
MozReview-Commit-ID: KmXRj8TBvYK
After this patch, the following will all display the subcommand help where they previously displayed
the command help:
$ mach help <command> <subcommand>
$ mach <command> --help <subcommand>
$ mach <command> <subcommand> --help
The command help will still be shown for:
$ mach help <command>
$ mach <command> --help
MozReview-Commit-ID: 7EsVblnCaFM
The sgr0 terminal capabity defines the byte sequence needed to reset
terminal text to its default state. For reasons documented inline in
this commit, we now print this sequence after every line printed by
mach's terminal logger.
MozReview-Commit-ID: 3RukP0QXtqy
This deprecates PYTHON_UNIT_TESTS and replaces it with PYTHON_UNITTEST_MANIFESTS.
In the build system, this means python unittests will be treated the same as all
other test suites that use manifestparser. New manifests called 'python.ini' have
been created for all test directories containing python unittests.
MozReview-Commit-ID: IBHG7Thif2D
The way it works now, `mach` commands often invoke subprocesses where
the subprocesses' stdio file descriptors are pipes so the mach command
can e.g. parse output.
Processes like clang, gcc, and cargo determine if they can send color
codes to {stderr, stdout} by seeing if those file descriptors are TTYs.
When e.g. `make` is executed via `mach`, this test fails because those
descriptors are pipes (even though they eventually end up on a TTY).
We can't wire the file descriptors to the TTY because `mach` needs
to analyze output. We don't want users defining process flags to force
color in their mozconfigs because color codes would still be sent
if stdout was not a TTY.
This patch sets the MACH_STDOUT_ISATTY environment variable in all mach
commands when stdout is a TTY. Subsequent processes can then look for
this variable to determine whether to override color settings, print
terminal control codes, etc.
MozReview-Commit-ID: GxXP2mQssjC
Currently, environment variables set when running mach commands will
propagate after the command is finished. This can allow unwanted state
to bleed through.
This likely isn't an issue today, but isolating state during code
execution is generally a good practice. So do that.
MozReview-Commit-ID: AdaomGub5EF
These config options can be defined in ~/.mozbuild/machrc or topsrcdir/machrc.
Aliases work similar to the identically named option in an hgrc.
For example:
[alias]
browser-test = mochitest -f browser
mochitest = mochitest -f plain
MozReview-Commit-ID: CnOocEslRUI
Runtime configs have been implemented for awhile, but disabled. This patch
enables configuration. Config files will be loaded in the following order
(later files override earlier ones):
1a. $MACHRC
1b. $MOZBUILD_STATE_PATH/machrc (if $MACHRC is unset)
2. topsrcdir/machrc
3. CLI via --settings
Note: .machrc may be used instead of machrc if desired.
MozReview-Commit-ID: IntONAZLGML
This adds a |mach settings locale-gen| subcommand to automatically generate locale
specific documentation for settings. It also refactors |mach settings-create| to
|mach settings| and moves |mach settings| to |mach settings -l|. Finally it performs
some misc cleanup mostly related to locales.
MozReview-Commit-ID: 1VWLcb9ehAH
Some sections should support user-defined options. For example, in an [alias] section, the option names
are not well-defined, rather specified by the user. This patch allows user-defined option names for any
section that has a 'section.*' option defined. Even with 'section.*', option types are still well-defined.
MozReview-Commit-ID: L34W9v9Fy28
Defining settings was a little complicated. First it required overriding a '_register_settings'
method, and then it required making N calls to a second 'register_setting' method within that.
This patch simplifies the process of defining settings by only requiring a
'config_settings' attribute. This attribute should be a list of tuples:
[
('<section>.<option>', '<type>', <default>, set(<choices)),
]
`default` and `choices` are optional. Alternatively, 'config_settings' can be a callable
that returns a list of tuples in the same format. This still allows for greater flexibility
for more advanced cases.
MozReview-Commit-ID: F4DTTNJdJsa
For example, say there is a command 'foo' that has a subcommand 'bar'. Prior to this, it was not
possible to run:
./mach foo
as its own independent command. The above would instead print the subcommand help for 'bar'.
MozReview-Commit-ID: JU4dXoxnCyu
Ted Mielczarek