In bug 1637665, we changed the OS X codepath for `mach run` to consult
`MOZ_WIDGET_TOOLKIT` as a substitute for "browserness". Unfortunately,
this caused tests that only set `MOZ_MACBUNDLE_NAME` to fail. Rather
than set `MOZ_WIDGET_TOOLKIT` in those tests, we probably should have
been checking `MOZ_MACBUNDLE_NAME` in the first place, since
`MOZ_MACBUNDLE_NAME` is also set only for the browser and not for the JS
shell. We can also remove the xulrunner codepath, as xulrunner was
removed from the tree several years ago.
Differential Revision: https://phabricator.services.mozilla.com/D75315
In Bug1127565 bootstrap was given support for the openSUSE Linux distribution
Fetching the Stackwalk toolchain was not added unfortunately
his bug1636797 was hit during testing...
(also fixed some formatting "issues")
Differential Revision: https://phabricator.services.mozilla.com/D75189
We need to skip the xulrunner/browser code (do we even support xulrunner
anymore?) for `--enable-application=js` builds. `MOZ_WIDGET_TOOLKIT` is
used elsewhere to check for "browserness".
Differential Revision: https://phabricator.services.mozilla.com/D75140
The official decision appears to be that we want people to ask questions in the build channel on chat.mozilla.org for queries that require build peer approval, as opposed to emailing specific people directly, filing bugs, etc. Rather than the vague "consult a build peer" suggestion currently in the code, specify exactly what we expect people to do in mozbuild.
Differential Revision: https://phabricator.services.mozilla.com/D74963
This patch is adding an option to push a perftest run in the CI.
It's based on :
- sparse profiles
- push_to_try
- options passed through try_task_config.json
Differential Revision: https://phabricator.services.mozilla.com/D74115
This patch enables perfherder data validation with jsonschema and also adds some extra tests for the Perfherder layer.
Differential Revision: https://phabricator.services.mozilla.com/D74877
`hg` gives us some additional levers to tweak when determining which commits are public and non-public. All the better, because `hg`'s default behavior under a rebase is to preserve the original commit time, so if you rebase a month-old commit on the latest version of central, you might still get the "revision too old" error despite your tree actually being completely up-to-date. Here we replace the original logic with something a little bit more robust to that scenario.
As before, this is a best-effort check that might have false positives or false negatives; this commit doesn't change that, but hopefully it decreases the amount of false positives under `hg`.
Differential Revision: https://phabricator.services.mozilla.com/D74745
The current error message leaves you with basically no recourse besides filing a bug if you're already at the latest HEAD. Meanwhile, `mach clobber` will fix it but in doing so you're taking a very blunt sledgehammer to the problem. Instead, I've updated this error message to tell you you can `mach clobber python`. I also removed the explicit reference to "artifact builds" because you can encounter this error outside of artifact builds as well. Finally, I added another reminder that `mach bootstrap` and `mach artifact` don't work for old revisions of central because I keep getting bugs about it and more screaming about how it's unsupported can't hurt.
Differential Revision: https://phabricator.services.mozilla.com/D74732
|./mach try| subcommands are now compatible with both python 2 and 3.
Hand-tested with many combinations of subcommand and subcommand flags.
Updates tryselect unit tests to use Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D73398
Adds support for all variations of fenix (app name and activity passed by argument to mach perfest)
Fixes bug in mach perftest argument passing (android-activity was not being set)
Differential Revision: https://phabricator.services.mozilla.com/D73334
Install minidump_stackwalk as part of 'mach bootstrap' so that it is readily available
for generating crash reports, if desired.
Differential Revision: https://phabricator.services.mozilla.com/D74442
Python 3 doesn't allow strings to be written to files opened in binary mode
(it requires a byte array in that case). As it happens, we should really be
opening these Eclipse config files in text mode since it seems on Windows the
files use Windows line ending characters. So rather than change the strings
to byte arrays, this patch simply changes the code to open the files in text
mode.
Differential Revision: https://phabricator.services.mozilla.com/D74318
When handling bug 1632429, I found some tests that worked on Python 2, but not Python 3.
They were marked accordingly as "expected failures". However, my system version of Python
is 3.8, while CI (and a non-trivial number of devs, probably) use 3.6.
Some of these tests marked as xfail were actually still working on versions of Python until 3.8.
The failure of this test was due to a change in default tarfile format. Explicitly setting this
format makes the tests pass in all relevant python versions.
Differential Revision: https://phabricator.services.mozilla.com/D74337
`ply`, [by design](https://github.com/dabeaz/ply/issues/79), does not produce reproducible table files; hence bug 1633156. (Note that this was *always* true, but only became a problem once we switched to Python 3, which has more unpredictable dict iteration order than Python 2.7, at least prior to [3.7](https://docs.python.org/3/whatsnew/3.7.html#summary-release-highlights).)
In any other circumstance I would consider submitting a patch to `ply` to fix this, but as of the [in-progress version 4.0 of the library](https://github.com/dabeaz/ply/blob/master/CHANGES), it doesn't even emit this cached data any more, and indeed the [latest version of the code](1fac9fed64/ply) doesn't even call `open()` at all except to do logging or to read the text data to be parsed from `stdin`. So if we were going to pin our future on `ply` and upgrade to later versions of the library in the future, we would have to live in a world where `ply` doesn't generate cached table files for us anyway.
Emitting the cached table files so later build steps can consume them is an "optimization", but it's not clear exactly how much actual value that optimization provides overall. Quoth the `CHANGES` file from that repository:
```
PLY no longer writes cached table files. Honestly, the use of
the cached files made more sense when I was developing PLY on
my 200Mhz PC in 2001. It's not as much as an issue now. For small
to medium sized grammars, PLY should be almost instantaneous.
```
In practice, I have found this to be true; namely, `./mach build pre-export export` takes just about as long on my machine after this patch as it did before, and in a try push I performed, there's no noticeable performance regression from applying this patch. In local testing I also found that generating the LALR tables in calls to `yacc()` takes about 0.01s on my machine generally, and we generate these tables a couple dozen times total over the course of the `export` tier now. This isn't *nothing*, but in my opinion it's also not nearly long enough where it would be a concern given how long `export` already takes.
That `CHANGES` file also stresses that if caching this data is important, we have the option of doing so via `pickle`. If and when we decide that re-enabling this optimization is valuable for us, we should take control of this process and perform the generation in such a way that we can guarantee reproducibility.
Differential Revision: https://phabricator.services.mozilla.com/D73484
We always print the output when there is no error. In case of error, we
stick the output in the thrown exception, but nothing actually prints
that out. It's simpler to just let the subprocess print its own output
instead of capturing it, so that important error messages are not hidden
in the case of failure.
Differential Revision: https://phabricator.services.mozilla.com/D74004
We currently generate absolute paths in all of our XPIDL-generated
source files, which is not so great for several reasons (deterministic
generation of files across machines, Searchfox analysis logic, shared
compilation caches, etc.). Let's generate paths that still indicate
where you should be looking, but are identical across compilations,
objdirs, etc.
Differential Revision: https://phabricator.services.mozilla.com/D73747
Combined with the upcoming upgrade of the build environment to Debian 8,
which is also going to upgrade the Gtk+3 requirement to 3.14, of the
major Linux distros Firefox currently supports running on, this removes
runtime support for:
- Fedora 20 and earlier (EOLed in 2015),
- OpenSUSE 13.1 and earlier (EOLed in 2016),
- Debian 7 (EOLed in 2018),
- Ubuntu 14.04 (EOLed in 2019).
Some of them might actually be supported in practice because the Gtk+3
requirement might be more relaxed than 3.14, but figuring that out is not
worth the effort.
Differential Revision: https://phabricator.services.mozilla.com/D73783
We currently check that the binaries we ship are not using symbol
versions of system libraries that would not be available on some older
systems. In some cases, however, we may get dependencies on newer symbol
versions in the form of weak symbols, that are checked for at runtime.
This happens with __cxa_thread_atexit_impl when building against a glibc
newer than 2.18, and the supporting code in Rust libstd actually checks
at runtime whether the weak symbol is resolved before using it.
Differential Revision: https://phabricator.services.mozilla.com/D73782
We currently generate absolute paths in all of our XPIDL-generated
source files, which is not so great for several reasons (deterministic
generation of files across machines, Searchfox analysis logic, shared
compilation caches, etc.). Let's generate paths that still indicate
where you should be looking, but are identical across compilations,
objdirs, etc.
Differential Revision: https://phabricator.services.mozilla.com/D73747
We need to set up the notebook deps.
The train of patch re-activated some
tests that required the dependencies used
by the notebook, which were not added in the
path.
Differential Revision: https://phabricator.services.mozilla.com/D73936
It's not maintained and doesn't work as-is. If we want to revive it we can grab it from source control later, but in the meantime it's just confusing and it comes up in search results even though we never look for it.
Also delete `mach analyze` which depends on the existence of a Tup backend.
Differential Revision: https://phabricator.services.mozilla.com/D73911
It's not maintained and doesn't work as-is. If we want to revive it we can grab it from source control later, but in the meantime it's just confusing and it comes up in search results even though we never look for it.
Also delete `mach analyze` which depends on the existence of a Tup backend.
Differential Revision: https://phabricator.services.mozilla.com/D73911
This patch implements the new intermediate results standard and adds the mechanisms required to handle it. Results validation is done with jsonschema and some manual validation (because of some unfortunate issues with jsonschema) and some tests were implemented to ensure that we fail/pass where expected. The metrics modules were modified to handle multiple suites.
One thing that is disabled in this patch is the subtest/single-metric specifications through the "results" field. We'll do one thing at a time here and we also have no use for subtests yet (although we definitely will).
Differential Revision: https://phabricator.services.mozilla.com/D72067
`./mach bootstrap` prepares your environment for a build and installs missing "brew" packages".
However, if a package is installed but out-of-date, it was being ignored by the bootstrapping logic.
This change ensures that `brew update` is run, and updates any out-of-date packages that are requested.
Differential Revision: https://phabricator.services.mozilla.com/D72956
There are `conditions` in tree that are callables but which don't have a `__name__` attribute; for example, `functools.partial` instances don't have a `__name__` since they're effectively anonymous functions. If you get to this branch and one of your `conditions` are that kind of object then you'll get a confusing error message instead of the understandable one we're trying to produce here, so account for that possibility.
Differential Revision: https://phabricator.services.mozilla.com/D72957
This warning dates from bug 910487, which was 7 years ago. Since joining Mozilla I have *always* gotten this warning, and as far as I can tell since I never had a pre-2019 version of Visual Studio on my dev machine, the VS90COMNTOOLS variable was *never* set. Moreover, the "hack" is written in such a way that it does nothing *unless* you have `VS{100,110,120}COMNTOOLS` set, which I never have on my machine either, as you might expect since I only have the one version of Visual Studio installed.
The [latest public build documentation](https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Windows_Prerequisites) recommends that you install the Community edition of Visual Studio 2019, and as of 2019 the variable that's being used is `VS160COMNTOOLS`, so the only way someone would get value out of this hack is if they're using a substantially older version of Visual Studio than we recommend anyway.
Since 1) I *suspect* the hack is not doing anything for the large majority, if not all, of the people currently running builds on Windows on a day-to-day basis and 2) even if the hack continues to do something useful under some hypothetical scenarios, the content of the hack as well as the corresponding warning is so outdated that it should be updated anyway, I propose deleting it entirely.
Differential Revision: https://phabricator.services.mozilla.com/D72925
This patch adds support for tests metadata. A test script parser is added as
well as a new "doc" flavor that can be used to display the script info in the
command line. This parser will be the basis for building automated docs and
scripts verifications if we want to do this.
Differential Revision: https://phabricator.services.mozilla.com/D72800
The current way to get outgoing files in Git passes every branch to merge-base, which generates a way too long command that fails on Windows. This patch tries using `git log --not --remotes` instead of passing merge-base value.
Differential Revision: https://phabricator.services.mozilla.com/D71802
pip-compile can annotate each requirement with a reason why it is included (e.g.: transitive dependency, or depended-on directly).
When annotating direct dependencies, it notes it as "via -r <direct path to requirements.in file>".
Since we were using a temporary directory, the path of the directory was being included, making the output non-deterministic.
This change ensures that we run pip-compile in the same working directory as the temporary requirements file, enabling
the annotations to be deterministic: "via -r requirements-mach-vendor-python.in".
Differential Revision: https://phabricator.services.mozilla.com/D72181
`os.path.realpath` in Python 3.8 now always uppercases Windows drive letter, while it was just an alias of `os.path.abspath` in Windows. This patch uses `.realpath()` consistently to get `topobjdir` to fix the incompatibility from the behavior change.
Differential Revision: https://phabricator.services.mozilla.com/D72188
Split out jsreftest and jittest files into their own packages, removing them
from the common package.
This speeds up extracting files from the common test archive for
non-jsreftest/jittest suites.
Also, remove some files from the web-platform test archive that are already
present in the common archive.
Differential Revision: https://phabricator.services.mozilla.com/D72192
As we migrate from makefiles to moz.build, we want to warn developers if
variables only used by moz.build are accidentally defined in a makefile.
This was the purpose of `MOZBUILD_VARIABLES` in recursivemake.py, though it became out-of-date.
This patch defines `MOZBUILD_VARIABLES` off the official list from `mozbuild.frontend.context.VARIABLES`, and
removes unused (?) code from makefiles accordingly
Note that the following variables use to be in `MOZBUILD_VARIABLES`, but aren't there any more because
they aren't in `mozbuild.frontend.context.VARIABLES`:
* CMSRCS
* CMMSRCS
* EXTRA_JS_MODULES
* EXTRA_PP_COMPONENTS
* EXTRA_PP_JS_MODULES
* HOST_CSRCS
* HOST_CMMSRCS
* HOST_EXTRA_LIBS
* JAVA_JAR_TARGETS
* LIBS
* MAKE_FRAMEWORK
* MODULE
* NO_DIST_INSTALL
* NO_INTERFACES_MANIFEST
* PARALLEL_DIRS
* PREF_JS_EXPORTS
* RESOURCE_FILES
* SHARED_LIBRARY_LIBS
* TEST_DIRS
* TOOL_DIRS
* XPCSHELL_TESTS
Differential Revision: https://phabricator.services.mozilla.com/D72076
As we migrate from makefiles to moz.build, we want to warn developers if
variables only used by moz.build are accidentally defined in a makefile.
This was the purpose of `MOZBUILD_VARIABLES` in recursivemake.py, though it became out-of-date.
This patch defines `MOZBUILD_VARIABLES` off the official list from `mozbuild.frontend.context.VARIABLES`, and
removes unused (?) code from makefiles accordingly
Note that the following variables use to be in `MOZBUILD_VARIABLES`, but aren't there any more because
they aren't in `mozbuild.frontend.context.VARIABLES`:
* CMSRCS
* CMMSRCS
* EXTRA_JS_MODULES
* EXTRA_PP_COMPONENTS
* EXTRA_PP_JS_MODULES
* HOST_CSRCS
* HOST_CMMSRCS
* HOST_EXTRA_LIBS
* JAVA_JAR_TARGETS
* LIBS
* MAKE_FRAMEWORK
* MODULE
* NO_DIST_INSTALL
* NO_INTERFACES_MANIFEST
* PARALLEL_DIRS
* PREF_JS_EXPORTS
* RESOURCE_FILES
* SHARED_LIBRARY_LIBS
* TEST_DIRS
* TOOL_DIRS
* XPCSHELL_TESTS
Differential Revision: https://phabricator.services.mozilla.com/D72076
When running e.g. check_symbols with extra flags like when checking
for vpx_codec_dec_init_ver when building against system libvpx, in some
configurations, the test can fail when the library flags (-l) appear
before the source file path.
The reason is that in some configurations, the compiler passes
--as-needed to the linker before both the flags and the object file
path, and the object file path is in the same position as the source
file path was. With --as-needed, -l flags are dropped if the library
wasn't needed for any of the linked code that appears *before* the flag.
So linking with `--as-needed -lfoo foo.o`, is equivalent to linking with
`foo.o` only in practice, while `--as-needed foo.o -lfoo` is equivalent
to `foo.o -lfoo`.
Differential Revision: https://phabricator.services.mozilla.com/D71456
We're back to an explicit --android option and --android-app-name, the layer won't use the --browser-binary option as an implicit
fallback anymore. This is cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D72031
