It is useful to be able, during mozconfig execution, to do tests depending
on what was previously added with mk_add_options. Specifically, there is a
need to do this for MOZ_PGO because developers pushing to try may add it to
mozconfig.common.override.
While, ideally, it would be nice if we just defined the variable itself in
the mozconfig execution environment, that is a tedious task, having to jump
through hoops with eval, and handle all cases of variable assigment properly.
The hacky alternative is to just treat MOZ_PGO specially, but meh.
So instead, we set a ${var}_IS_SET variable to 1, indicating that a
mk_add_options defined ${var} to some value.
Bug 1153566 changed the regexp used in that method in such a way that there
has been a big hit in the time spent executing the make backend. On my machine,
with the current code, mach build-backend indicates:
Backend executed in 5.01s
With the change from bug 1153566 reverted:
Backend executed in 2.97s
That's a significant regression for a 4-character change.
But we can actually avoid using regexp in most cases, which can make things faster
than they were. With this change, we get down to:
Backend executed in 2.28s
For reference, making the _check_blacklisted_variables method do nothing at all
ends with:
Backend executed in 2.20s
DONTBUILD NPOTB on a CLOSED TREE
--HG--
extra : commitid : 2ft26PqDgyT
extra : rebase_source : ca2eb4fe7c04e97fb78cd2175bd8549e13c13d7e
extra : amend_source : 10a2fa5931d151af448269239bdf2dc1980d3fae
This just allows a little versatility for consumers such as b2gdroid,
which are Fennec-like but don't have MOZ_APP_NAME=fennec.
I elected to pass appname as a parameter rather than modify the
existing distdir because I expect to want to differentiate, in some
way, the output AAR files based on the underlying name. That is, in
future we might generate geckoview-fennec-VERSION.aar and
geckolibs-b2gdroid-VERSION.aar, or stuff the name into the Ivy version
information, or...
This also fixes a typo in one of the JarFinder instantiations.
--HG--
extra : commitid : CnJKouGgkh1
extra : rebase_source : 5767e66ea53e14dd6468adec741773a02a6e2d3a
CalledProcessError.output and subprocess.check_output's return value
are str types. This file uses unicode_literals. If we do something
like `if 'foo' in e.output`, there will be a mix of str and unicode
types and Python will do implicit conversion. If the strings aren't
ASCII, we'll likely encounter a UnicodeDecodeError.
Use b'' literals around all strings to prevent this coercion from
occurring.
--HG--
extra : commitid : 89l5oW9T8ks
extra : rebase_source : f402353cd4a759c882475518f1360687af1936dc
extra : amend_source : ec31a39620eae2c90fb832f314cd161905444a34
This will give us the ability to execute custom code at command dispatch
time. We can use this for global tests before dispatch.
--HG--
extra : commitid : 5vfmSqOis4W
extra : rebase_source : c285b16007c4b604b164d079a275198c8760e480
I considered three ways to do this:
* one, as a Python script executed with $(shell);
* two, as a Python script that writes an include file for the preprocessor;
* three, as a function exposed to the moz.build sandbox.
I rejected two because it's both tied to the preprocessor, and awkward
to make handle the dependency on the buildid (in a file) and
additional build defines (in config.status).
I rejected three because I know of no precedent for this approach, and
it hides the dependency on the buildid.
One doesn't handle failures in the script gracefully, but neither did
the existing approach. This patch is at least testable.
--HG--
extra : commitid : 8dfw1ri7qjr
extra : rebase_source : da0e5ec705e0ac4c795bd2d7892f73857a1699ac
Without this, we throw UnrecognizedArgumentError when running commands
such as `./mach mochitest-plain test`, which causes an error message
such as the below to be emitted:
It looks like you passed an unrecognized argument into mach.
The mochitest-plain command does not accept the arguments: test
This patch will fix the above command to instead print the corresponding
deprecation message.
DONTBUILD NPOTB
This was just an oversight during the initial landing, leading to two
copies of artifact libraries being appended to the same destination
file.
--HG--
extra : commitid : JvvZDrjUOQZ
extra : rebase_source : f48c1a9d4b506b6ed931043aeeaca437418ea9c3
extra : histedit_source : 3dbfd049039c0adc595c1abb0df3dca3af9db207
This implements a new "scan" mode for DMD that records the address
and contents of every live unsampled block in the DMD log. This
enables the low-level analysis of references from one block to
another, which can help leak investigations.
bf34d16b6ab2 added absolute_import to this file. When changed, "import
mozinfo" stopped picking up mozbuild.mozinfo and started importing
mozinfo instead.
Use relative imports to force mozbuild.mozinfo to be picked up.
--HG--
extra : commitid : 4GC5lJWrBFq
extra : rebase_source : c91674b16f29dadf8dcda460787ea94654f2864a
This is a regression from bug 1176620 that results in all Firefox mach commands showing up in the help for B2G, even though they don't work there.
--HG--
extra : rebase_source : e779b866a82c2df1dbf913d24b93a0dc9838ff9f
This was the only import of glob from all mach_commands.py files. Kill
it.
With this commit, there are no modules imported by a single
mach_commands.py outside of testing/web-platform/mach_commands.py.
--HG--
extra : commitid : 4CJqlwDqOVg
extra : rebase_source : 9dbbd69291d64b894a399523864562107c10872e
This import brought in significant parts of the mozbuild package. Moving
it to a deferred import reduces the total number of Python modules
imported during mach dispatch by 43.
--HG--
extra : commitid : GdXsF7agvCT
extra : rebase_source : 586f1960c8e7eb400f61467045b064167784f68b
This removes ambiguity as to which modules are being imported, making
import slightly faster as Python doesn't need to test so many
directories for file presence.
All files should already be using absolute imports because mach command
modules aren't imported to the package they belong to: they instead
belong to the "mach" package. So relative imports shouldn't have been
used.
--HG--
extra : commitid : 6tFME1KKfTD
extra : rebase_source : 78728f82f5487281620e00c2a8004cd5e1968087
DONTBUILD ON A CLOSED TREE: Android-only and the build changes are cosmetic.
Very much a first cut, but I'd like to get some Fennec early adopters testing.
This adds:
* |mach artifact install| to fetch and install Fennec binaries;
* |mach artifact last| to print details about what was last installed;
* |mach artifact {print,clear}-caches|, for debugging.
This code is exposed as a new mozbuild.artifacts Python but it's not
particularly general. My intention was to get things out of the mach command
more than produce a general artifact fetching API. We can leave that bike
shed to Bug 1124378.
I've been testing this with --disable-compile-environment and it works well
locally, although there's no reason a knowledgeable developer couldn't use
this in conjunction with a fully-built tree. (I don't know when such a
situation would arise, but I know of no technical impediment.)
--HG--
extra : commitid : 1T28aVfArqF
extra : rebase_source : b8c11244de8be0a14d605853f30cd47312d0a4ba
extra : histedit_source : 78a224501cd3cf0f86707c9c9549b61b4b248ba7
This type is now redundant with our new rich type for capturing all mach
command metadata. Eliminate it and using _MachCommand instead.
--HG--
extra : commitid : 7pXhf6V7F8m
extra : rebase_source : 343615096f23d3acf23f7487c7b8c7137c85337e
Simplify construction of mach's MethodHandler instances by by passing in
our new rich type that holds all command metadata.
While we are here, kill the docstring argument, as it can be computed
easily inside MethodHandler.__init__.
--HG--
extra : commitid : I5PRlYDVXVq
extra : rebase_source : ebf07116357036ddfec06fd32fb161fefff628e6
Up to this point, mach command metadata has been stored in tuples.
Initially, things weren't so bad. But they have evolved into tuples with
many elements. Adding new attributes is cumbersome. Let's restructure
the code to capture metadata in a dedicated class.
Before, there existed a separate attribute on the @Command or
@SubCommand decorated method for each mach decorator: @Command,
@CommandArgument, @CommandArgumentGroup. With the magic of __ior__,
we can now capture all metadata on a single type. This simplies
processing, as we now only look at a single attribute on methods:
_mach_command.
Before, we used separate attributes to distinguish between mach commands
and mach sub-commands. Now that we have a type that can hold all data,
we combine things into the _mach_command attribute and look for the
presence of the "subcommand" attribute on this type to identify
sub-commands.
--HG--
extra : commitid : EtBwUmS5TV2
extra : rebase_source : 4ef5f95a532693672c6c4b33fa1c22adb76d105e
We want this logic to live next to where metadata types are defined so
downstream consumers of Files-based metadata don't have to reinvent the
wheel.
The work in this commit will be used to enable auto filing bugs during
pushes to MozReview.
--HG--
extra : commitid : F55RzhDCVmR
extra : rebase_source : 1fe1a03f8d592c60d2ed8d760fd62c7bc60421a5
This file hasn't been updated in ages and the current configuration
doesn't produce working packages. Change that.
--HG--
extra : commitid : AfiwohniLPq
extra : rebase_source : 70786c7bd9e37c9acc9b38f54c956fec424cfcf5
extra : amend_source : e4e3752dc97b934456b973736ef55633366c8ee6
This should have been done in 4c757e339f81.
DONTBUILD (NPOTB)
--HG--
extra : commitid : 41TCyM9iqVQ
extra : rebase_source : 8defc6b23d507e1580db77869ff2c2e99ee5db5a
extra : amend_source : 5f330ea7c2775675fcc8051e5c46d2439c4e5823
CentOS has apparently changed its disto name. Detect the new version.
DONTBUILD (NPOTB)
--HG--
extra : commitid : JrYp4WJ5Fs3
extra : amend_source : c2ac16f7006c2e44450de54edb732fa722f150d0
The hglib Mercurial finder was nice. But it is somewhat slow, as it
involves a separate Mercurial process.
This commit introduces a native Mercurial finder that speaks directly
to a Mercurial repository instance. It is significantly faster.
The new code is isolated to its own file because it imports Mercurial
code, which is GPL.
--HG--
extra : commitid : 8CzDFt3lQmx
extra : rebase_source : 1540ad20d795086926b316612062e2a1f10c4958
This commit adds support for specifying a Mercurial revision with `mach
file-info`. Aside from being a potentially useful feature, it proves
that MercurialRevisionFinder works with BuildReader.
--HG--
extra : commitid : 7143XT9ENqb
extra : rebase_source : d8c0c98d536d07db76653b648fd4b7d74d8f43ae
The moz.build reader uses absolute paths when referencing moz.build
files. *Finder classes expect paths to be relative to a base. When we
switched the reader to use *Finder instances for I/O, we simply provided
a default FileFinder based at the filesystem root. This was quick and
easy. Things worked.
Unfortunately, that solution isn't ideal because not all *Finder
instances can accept absolute paths like that. The
MercurialRevisionFinder is one of them.
Changing the moz.build reader to talk in terms of relative paths is a
lot of work. While this would be ideal, it is significantly easier to
defer the problem until later and hack up MercurialRevisionFinder to
accept absolute paths. This commit does exactly that.
Bug 1171069 has been filed to track converting BuildReader to relative
paths.
--HG--
extra : commitid : 2JmzOBldBLy
extra : rebase_source : a8af6ce88dd9e2b98f131c92b45c3ece852e13d2
Now that moz.build files use finders for I/O, we can start reading
moz.build data from other sources.
Version control is essentially a filesystem. We implement a finder
that speaks to Mercurial to obtain file data. It is able to obtain
file data from a specific revision in the repository.
We use the hglib package (which uses the Mercurial command server) for
speaking with Mercurial. This adds overhead compared to consuming the
raw Mercurial APIs. However, it also avoids GPL side-effects of
importing Mercurial's Python modules.
Testing shows that performance is good but not great. A follow-up
commit will introduce a GPL licensed Mercurial finder. For now, get
the base functionality in place.
--HG--
extra : commitid : BkwaQOW9MiR
extra : rebase_source : 915d6015317ccc79c228a76eed861d9f43e2fd17
Sometimes moz.build data may not come from the local filesystem. To
support defining moz.build data in other backing stores, we switch
moz.build reading to use mozpack's *Finder classes for I/O.
The use of a FileFinder bound to / is sub-optimal. We should ideally
be creating a finder bound to topsrcdir. However, doing this would
require refactoring a lot of path handling in the reader, as that code
makes many assumptions that paths are absolute. This would be excellent
follow-up work.
While I was here, I cleaned up some linter warnings for unused imports.
On my machine, this commit results in a slight slowdown of moz.build
reading. Before, `mach build-backend` was consistently reporting 1.99s
for reading 2,572 moz.build files. After, it is consistently reporting
2.07s, an increase of 4%. This is likely due to a) function call
overhead b) the cost of instantiating a few thousand File instances
c) FileFinder performing an os.path.exists() before returning File
instances. I believe the regression is tolerable.
--HG--
extra : commitid : 1WDcrSU3SQD
extra : rebase_source : a23aa5a4063c6f7080ee00b6f0fe0ee532da3ce7
Passing raw file handles around is a bit dangerous because it could lead
to leaking file descriptors. Add a read() method that handles the simple
case of obtaining the full contents of a File instance.
This should ideally be a method on BaseFile. But this would require
extra work and isn't needed. So we've deferred it until bug 1170329.
--HG--
extra : commitid : 82qw76XmpjC
extra : rebase_source : 422b16c5a3b1577f080097925aeaeb560aa3e798
Today, the *Finder classes are optimized for doing matching and
returning multiple results. However, sometimes only looking for a
single file is wanted.
This commit implements the "get" method on all the *Finder classes.
It returns a BaseFile or None.
FileFinder._find_file has been refactored into FileFinder.get
to avoid code duplication.
--HG--
extra : commitid : K9It2ZJ3Rbo
extra : rebase_source : a56f8f70aa1902d26373a7196eae2847cce653d3
The current mode of execution of templates doesn't allow them to more advanced
control flow, like returning early, returning or yielding values, because that
mode of execution is equivalent to running the code at the top level of a .py
file.
Making the templates executed through a function call, although trickier,
allows those control flows, which will be useful for template as context
managers.
inspect.getsourcelines() and inspect.getfile() involve I/O out of our control.
Our use of those functions, however, doesn't require all their smarts. In fact,
we only use them on function objects, for which we can just do the work
ourselves without involving inspect functions that trigger I/O.
`mach help <command>` currently only displays a brief description of the
command along with its arguments. Sometimes more detailed help text is
needed.
With this commit, the docstrings of mach command handlers will appear in
the output of `mach help <command>` if they are defined.
I've implemented basic docstrings for the three flavors of mach commands
(normal command, main subcommand, subcommand) to demonstate things work.
My hope is others will start to fill in docstrings once this feature
lands so the output for `mach help` can serve as a better learning guide
for new contributors.
--HG--
extra : commitid : Hx6ZkHDxbCK
extra : rebase_source : 01ced5a044442e370a45cd3fb245ac6283316925
extra : amend_source : fceb771e0e1ffa4e6f3f1b7c22eae6e25cf82034
The old way of writing scripts for generated files would invoke the script thusly:
python script.py arg1...
Invoking the script this way means that the script's directory is
automatically added to sys.path, and importing modules from that
directory is easy. Let's make it equally easy in the new world for
GENERATED_FILES, too.
Document the role of l10n.ini, filter.py, file paths and checks.
Also add a glossary for the jargon used in the doc.
--HG--
extra : rebase_source : ec71f9b7123ba76370d34d2349b2f078f14dd1de
Add generic support for different forms of paths in moz.build:
'/topsrcdir/relative/paths'
'srcdir/relative/paths'
'!/topobjdir/relative/paths'
'!objdir/relative/paths'
This drops the use of UserString for performance reasons, which required
going around with a meta class.
This code has been commented since it landed, and enabling it requires
adding the proper conditionals around all non_unified_sources in gyp
configurations, which is a daunting task. OTOH, the commented code is
already outdated (it would need updates to work) and the related code
that is not commented gets in the way of upcoming changes, so remove
it.
This helps upcoming changes, and relieves backends from path resolution.
This has the side effect of chaning the order of some unified sources,
which consequently breaks nsTextFormatter because it declares snprintf
methods after nsStringAPI #defines it.
A host elfhack binary is only built when there is a compile environment.
--HG--
extra : rebase_source : 4f1da429c581dfd81cbe3d5164c7586066cf6e79
extra : amend_source : f26fe7f3b44291f38459a81b9ff31bd6dbd220aa
extra : histedit_source : 93c628a500c66c46d522bfe678500bf5b5bf0de9
With TemplateContexts keeping the name of the associated template, and the
Gyp context being declared as a TemplateContext, it is now possible to know
the equivalent of IS_GYP_DIR just by looking at the template name.
Add .rs as a recognized file extension in SOURCES.
Propagate that through to the Makefile backend and add a dependency
generated and an explicit rule to call $(RUSTC) to compile them.
rustc builds static libraries, not obj files. At least, if one
asks it to output an obj file, I'm not clear how to get all the
compiler-specific runtime libraries the code will expect to link
to. Therefore we generate a static library for each rust source
file (which must be a complete crate for the time being) and link
that. Because of the extension it ends up on the LIBS line in the
the corresponding .desc file.
Note that the static library does still depend on some system
libraries, e.g. -ldl -lpthread -lm on linux. Gecko already
links to all of those, so we don't keep track of it here.
Should we need to add explicit linkage for other targets,
rustc does print a list to stderr which can be parsed.
--HG--
extra : rebase_source : 9f66101fd15a649a952d5481cf9561416204272a
Formerly, running |./mach test image/| would result in running a number of devtools tests
in addition to running tests under the image/ subdirectory. With this change, only tests
under the image/ directory would be run. Note that ./mach test animations or similar will
still run a variety of tests across the tree, because this input does not match a directory.
--HG--
extra : rebase_source : 8a0482796ccb51cec8cd873cf8cd9933e30d46d1
mach dispatch makes separate, independent calls to construct build
system state. Part of this resolution is determining the object
directory. For environments without an object directory defined, we must
execute config.guess to determine the object directory. This redundant
execution of config.guess can result in significant execution overhead.
Before this patch, `mach help` with no mozconfig took ~1.5s on my OS X
machine. After this patch, it goes down to ~0.750s. On Windows, the
difference is even more pronounced, with execution time dropping from
8.5s to 0.930s.
--HG--
extra : rebase_source : e8c0f49ceb71e1371e174e31f6e1f354d1e51b65
extra : amend_source : 133b013d5dd8f3817f92555aed1717534769585d
`mercurial version` doesn't always print the version number. `mercurial
--version` does. Use the latter.
--HG--
extra : rebase_source : 6f3c305ce8b38599a60c01d9cee410b724bfc37f
extra : amend_source : 430a3cf66094a62fabee59dbf4a486d4392a8c47
Now that moz.build can see EXTRA_*COMPONENTS and NO_JS_MANIFEST, we can
move some logic from rules.mk (executed every build) to moz.build's
emitter.py (executed only at build-backend time).
Interactive prompts make automation more difficult for developers looking to stand up machines using bootstrap en masse.
Two new options have been added to allow users with such needs to bypass all prompts: one for selecing an application
(desktop/mobile) and another for assuming yes to all questions posed by package managers (apt-get/yum).
This avoids duplicating the logic from SimplePackager to find base
directories, and fixes some cases where the l10n repack code wouldn't
find them properly.
