"distutils" builds packages to fit the architectures of the active
python.
However, the system Mac SDK doesn't always support all the same
architectures as the local Python binary.
To resolve this, we explicitly influence the build to only compile for
the specific architecture that the system runs on.
Assumes that "platform.machine()" will always line up with the "-arch"
flag of compilers.
Differential Revision: https://phabricator.services.mozilla.com/D96276
Previously, configure checked for Java binaries using
"configure"-specific "which()" and environment-variable-getter
functions. When bug 1670264 changed Java detection to use logic shared
between "configure" and "bootstrap", more general "which()"
/environment-variable-getter functions were used.
The test used to work because the "configure"-specific functions
respect the mocked-out file system and environment variables.
This patch adjusts the test to add mocks so the more general logic
is redirected to respect the mocked data.
Differential Revision: https://phabricator.services.mozilla.com/D96189
This adds back part of the code that was removed in bug 1339182,
reformats it with black, adjusts it to make flake8 happy, and converts
it to python 3.
This also adjusts the script after bug 1534003, which changed the
about:buildconfig page title.
Differential Revision: https://phabricator.services.mozilla.com/D95977
Before, on Windows, this resulted in installing the package in the parent environment (not the `virtualenv`). We fix this by passing down the `virtualenv_manager` so linters can install packages they need using that object's helper methods.
Differential Revision: https://phabricator.services.mozilla.com/D95738
With PYTHONPATH containing other directories, such as the python ones,
bad things can happen when the python script that is being run then
goes on to subprocess.Popen a python process for a different virtualenv,
or a different version, or whatever.
Differential Revision: https://phabricator.services.mozilla.com/D96155
Since python creates little-endian utf-16 consistently whether
cross-compiling from Linux or compiling natively on macOS, we could
write a small script that essentially replaces iconv. On the other hand,
we're also doing some manual preprocessing on the InfoPlist.strings.in
files, and we might as well use the preprocessor for that.
So, we augment the preprocessor to allow an explicit output encoding
other than utf-8, and use the preprocessor instead of `sed | iconv`.
Differential Revision: https://phabricator.services.mozilla.com/D96013
This patch will make sure that we select the right
wheels for numpy and scipy when `--visualmetrics` is used,
so we don't need to compile them.
It also make sure we don't install all those packages
unless the environment wants to use visual metrics.
Differential Revision: https://phabricator.services.mozilla.com/D91740
Here, "errors caused by local changes" means "errors whose stack traces contain a reference to a file that is in the set of files changed locally". This implementation is a trade-off:
1. This check will not catch issues caused transitively by changes to local files. For example, consider a function that has been updated and its return type changed in a backwards-incompatible way, whereas callers were not updated appropriately. This would likely manifest as a type error in the calling function after the callee has returned.
2. This check WILL catch issues that come from locally changed files where the cause of the error doesn't originate from those local changes. For example, consider a function that's been locally updated but is never called in the failing codepath; if an exception is thrown, it's not due to this local change, and we shouldn't filter it out.
There are conceivable improvements that we could apply to fix deficiency (1); for example, we could track imports recursively starting from the oldest frame in the stack trace and match on that set of imported files. Note this would not handle dynamic imports properly, and that this could exacerbate issue (2).
Issue (2) could conceivably be addressed by attempting to filter the actual local diffs down to changes that actually may be causing the error. This is difficult to do generally especially in light of Python's dynamism, but there mayb be conservative improvements that we could make in this space.
Overall, neither of the above caveats are deemed to be sufficiently concerning that this patch should be blocked as-is, and the current situation with our Sentry logs is unusable due to all the noise. This patch will probably have a substantial impact on that noise without incidentally filtering out too much signal.
Differential Revision: https://phabricator.services.mozilla.com/D95607
The original change was tested as working on try/try-c-c, but was done during
the window when bug 1673992 had been backed out on mozilla-central. As a result,
the mozharness.zip that was tested had the "mozharness/" top level directory and
everything worked.
Differential Revision: https://phabricator.services.mozilla.com/D95741
Moving comm-central's /mozharness directory to /testing/mozharness/configs so
that it is layed out the same as mozilla-central's. This also opens the door
for other possibilities such as /comm/testing/mozharness/scripts.
Differential Revision: https://phabricator.services.mozilla.com/D95373
Rather than putting a top-level "mozharness" directory in "mozharness.zip",
this puts the root of mozharness at the root of the archive. Then we extract
the archive into a "mozharness" directory. This way the path on disk ends up
being the same, but generic-worker won't attempt to modify the permissions of
files in the cwd.
Differential Revision: https://phabricator.services.mozilla.com/D95138
This environment variable can cause the module resolution to break
and have unexpected behavior while running mach bootstrap.
Differential Revision: https://phabricator.services.mozilla.com/D95185
Rather than putting a top-level "mozharness" directory in "mozharness.zip",
this puts the root of mozharness at the root of the archive. Then we extract
the archive into a "mozharness" directory. This way the path on disk ends up
being the same, but generic-worker won't attempt to modify the permissions of
files in the cwd.
Differential Revision: https://phabricator.services.mozilla.com/D95138
The definition of `patch_main()` has behavior that kicks in only on Windows and for Python 2. Unfortunately, not all of our `mach` commands have been migrated to Python 3, so this still matters.
Bug 1654103 replaced the single-quoted strings in this function with double-quoted strings. This should be fine, except that we call into `multiprocessing.forking.main()` with some monkey-patching that is meant to fix a Windows-specific bug (see bug 1316140). We don't do any clever serialization or anything here and we end up just passing that source to `multiprocessing.forking.main()` which aggregates that source code [dumbly](https://github.com/python/cpython/blob/2.7/Lib/multiprocessing/forking.py#L259), wrapping everything in double-quotes again and passing it to `_subprocess.CreateProcess()`, which ends up failing if the source contains strings formatted with double quotes.
We could revert bug 1654103 and exempt this file from linting, but that is overkill given that this file otherwise contains useful stuff. Instead we move everything to another file, exempt that file from linting, and update `util.py` accordingly.
Differential Revision: https://phabricator.services.mozilla.com/D94909
We add new metrics `distro` and `distro_version`. Their meaning varies based on the actual OS:
1. For Linux, the pair will be the name of the distribution and the distribution's version (e.g. `ubuntu`/`20.04`);
2. for macOS, the pair will be the string `macos` and the macOS version (e.g. `10.15.7`); and
3. for Windows, the pair will be the string (`windows`, `MAJOR.MINOR.BUILD`);
4. and for all other OS'es, the first will be the value of `sys.platform`, and the version string will be empty.
Differential Revision: https://phabricator.services.mozilla.com/D94781
These files were omitted from the original patch because reformatting them required some manual intervention in order to avoid breaking unit tests. Generally the `noqa` lines were already there and just needed to be moved from one line to another (due to the reformatting by `black`), but sometimes `black` saw fit to move a bunch of stuff all onto one line, requiring me to introduce new `noqa` lines.
Besides the autoformat by `black` and some manual fixups, this patch contains no other changes.
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94052
Depends on D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
These files were omitted from the original patch because reformatting them required some manual intervention in order to avoid breaking unit tests. Generally the `noqa` lines were already there and just needed to be moved from one line to another (due to the reformatting by `black`), but sometimes `black` saw fit to move a bunch of stuff all onto one line, requiring me to introduce new `noqa` lines.
Besides the autoformat by `black` and some manual fixups, this patch contains no other changes.
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94052
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
process_install_manifest now only prints the message (not the stack
trace) of ErrorMessage exceptions, and the error config has been updated
to consider such messages to have the "ERROR" severity.
Differential Revision: https://phabricator.services.mozilla.com/D93854
We used to have a complicated scheme to figure out the minimum supported
version of clang on OSX, based on some compiler feature, which wouldn't
allow to do other version checks further down the line.
The main blocker for better tests was to be able to distinguish between
Xcode clang and plain clang, which turns out to be possible with the
__apple_build_version__ define.
We still need to map versions manually, but it's better than the current
status quo.
Differential Revision: https://phabricator.services.mozilla.com/D94261
