In order to use StrictOrderingOnAppendListWithFlags instances in
mozbuild template functions, we need += to work correctly. This patch
implements extend and the associated functions (including +=),
disallowing some behaviour where convenient.
There's a subtle point hidden in the isinstance() tests: before this
patch, it was not easy to compare two
StrictOrderingOnAppendListWithFlags instances to see if they had the
*same* set of flags. That was because two instances may not have the
same class, and would only share the common
StrictOrderingOnAppendList, which isn't enough to infer the presence
of flags. To be slightly more clear, concrete instances will have
class StrictOrderingOnAppendListWithFlagsSpecialization (although
there are still multiple instances of that class) and all extend from
the unique class StrictOrderingOnAppendListWithFlags.
--HG--
extra : commitid : AMVDYt8khR
extra : rebase_source : 1ce0698691fc03fbdf6a976e92017c1d60bad15d
extra : histedit_source : 4812a565179fb4fac2e4b5cd89c4efe74e794dfa
DONTBUILD NPOTB
The top source directory configuration requires
mobile/android/gradle/m2repo/**, so it stays. There's no value
changing the location; it contains an Android-specific Gradle plugin.
We note the removal of |mach gradle-install| and point to the new
documentation.
--HG--
extra : commitid : 9Nhz2dnBIgY
extra : rebase_source : 32a2b8a92d57f963feac2bae28fed5a9f1b26f93
extra : amend_source : bf53a0b2d3d4ac0618bc82fe79914bdeaf1c1e0a
Both these files, are, after all, define files, like other CONFIGURE_DEFINE_FILES.
They only happen to have a special requirement for an expansion for all defines,
which doesn't need to happen through traditional preprocessing.
This change adds consistency in how configure-related headers are being handled.
This is needed to support hgwatchman.
--HG--
extra : commitid : 8D2A8YPNimB
extra : rebase_source : 7d5932aa049dfb352b93a87c2c8087dd7b324aab
extra : histedit_source : 9863189f265eca9e0b9363e13c59a7d55f5c633d
This change allows specifying objdir-relative paths in EXPORTS to enable
exporting entries from GENERATED_FILES. Objdir paths in EXPORTS that are
not in GENERATED_FILES will raise an exception.
Example:
```
EXPORTS += ['!g.h', 'f.h']
GENERATED_FILES += ['g.h']
```
Given the implementation, this should also work for FINAL_TARGET_FILES,
FINAL_TARGET_PP_FILES, and TESTING_FILES, but those are not well-tested.
This patch also renames the install manifest for '_tests' to match the
directory name for convenience in some code I refactored.
--HG--
extra : commitid : CwayzXtxv1O
extra : rebase_source : 5fb6f461fc740da9bce14bbdbfabdfe618af8803
Future improvements to process_install_manifest's --track option will require
adding data in the tracking dump that uses an install manifest form, and I don't
want e.g. switching branches or bisection to require to clobber in order to do the
right thing, so this change future-proofs the install manifest reader.
There are currently two operating modes for process_install_manifest:
- default, which removes any file in the destination directory that is not
in the install manifest
- --no-remove, which doesn't do the above.
While install manifests also have the ability to deal with files that may
be left in the destination directory some other way, that requires knowing the
list of those files in advance, which is not always possible.
For instance:
- with the FasterMake build backend, install manifests are split such that
there is one manifest per application of addon directory (to allow more
parallelism), which means there is one for dist/bin and one for several
of its sub-directories.
- With --disable-compile-environment combined with artefacts, the backends
are not aware of e.g. all the libraries and executables that end up in
dist/bin.
If we want to properly remove files when they are removed from moz.build
or jar.mn, we can't use --no-remove, but the alternative would remove those
files
So add an option that keeps a list of all the files that were installed as
part of processing the given install manifest(s). That information is simply
a dump of the install manifest, which, while it contains more information
than currently required, will allow to do smarter things in the future.
The default behavior for a FileCopier's copy is to remove all the files and
directories in the destination that aren't in its registry.
The remove_unaccounted argument can be passed as False to disable this
behavior.
This change adds another possibility, where remove_unaccounted may be a
FileRegistry, in which case only the files in that registry are removed.
This allows to e.g. only remove files that were copied from a previous
FileCopier.copy, leaving aside files that were in the destination for some
other reason.
Future improvements to process_install_manifest's --track option will require
adding data in the tracking dump that uses an install manifest form, and I don't
want e.g. switching branches or bisection to require to clobber in order to do the
right thing, so this change future-proofs the install manifest reader.
There are currently two operating modes for process_install_manifest:
- default, which removes any file in the destination directory that is not
in the install manifest
- --no-remove, which doesn't do the above.
While install manifests also have the ability to deal with files that may
be left in the destination directory some other way, that requires knowing the
list of those files in advance, which is not always possible.
For instance:
- with the FasterMake build backend, install manifests are split such that
there is one manifest per application of addon directory (to allow more
parallelism), which means there is one for dist/bin and one for several
of its sub-directories.
- With --disable-compile-environment combined with artefacts, the backends
are not aware of e.g. all the libraries and executables that end up in
dist/bin.
If we want to properly remove files when they are removed from moz.build
or jar.mn, we can't use --no-remove, but the alternative would remove those
files
So add an option that keeps a list of all the files that were installed as
part of processing the given install manifest(s). That information is simply
a dump of the install manifest, which, while it contains more information
than currently required, will allow to do smarter things in the future.
The default behavior for a FileCopier's copy is to remove all the files and
directories in the destination that aren't in its registry.
The remove_unaccounted argument can be passed as False to disable this
behavior.
This change adds another possibility, where remove_unaccounted may be a
FileRegistry, in which case only the files in that registry are removed.
This allows to e.g. only remove files that were copied from a previous
FileCopier.copy, leaving aside files that were in the destination for some
other reason.
The only use of BRANDING_FILES[...].source is in xulrunner/app/moz.build, for
the app.ico file.
This file has not been useful since the removal of the xpinstall-based
installer in bug 344236... 9 years ago.
Currently mach treats the first argument to eslint as the path and moves it to
the end of the arguments but this breaks usage like "mach eslint -f json browser".
It used to be necessary to change to the directory you wanted to lint but now
the .eslintignore is at the top level we just run from the top level. This means
the path argument doesn't need to be special anymore.
--HG--
extra : commitid : 5ozct0pVSC4
extra : rebase_source : 22132a240d8e6f4d099dbcdeb793958d7173e154
extra : amend_source : 2b9931b4283e1c84f699027e13eccc33fcdec978
The current implementation of HierarchicalStringList allows the following:
FOO.bar = [
'foo',
'bar',
]
while
FOO.bar += [
'foo',
'bar',
]
would be invalid because of the StrictOrderingOnAppendList enforcement.
It also allows to overwrite the entire list with a subsequent
FOO.bar = [
'baz',
]
while we've explicitly forbidden such things for every other list.
While in the vicinity, fix HierarchicalStringList._get_export_variable to not
call the HierarchicalStringList constructor uselessly.
This will allow a new kind of special variable where it is possible to do
FOO += ['bar']
All the current special variables are either strings (for which __setitem__ would
be called with a different string object), or a read-only dict (which doesn't
allow modifications).
We have many unit tests in the tree for some small parts of the build system
pipeline, but we don't have anything that resembles an end to end test, and we
kind of rely on the resulting Firefox not being broken by our changes.
With the Faster make backend growing, I want to ensure it produces the same
thing as the recursive make backend, at least for the parts it supports.
This adds some kind of test that allows to check that.
The test I'm about to add doesn't have XPIDL files, and that currently avoids
the FasterMake backend to run properly. Also, in the future, when the FasterMake
grows the ability to build C++ files, it should be possible to build Spidermonkey
with the FasterMake backend, but it doesn't have XPIDL files either.
This makes it clearer that really it's the same thing as FINAL_TARGET,
with preprocessing.
We still keep DIST_FILES in backend.mk because it's shorter and doesn't
really matter.
This new ChromeManifestEntry object type is generic and can hold any kind of
chrome manifest entry, but we currently only emit them for binary components.
References to sub-directory manifests is left to the backend, for now, until
all manifest entries are emitted by the frontend.
Ideally, we should properly make and shell quote everything we print out
in makefiles, but that's a can of worms I don't want to open just yet. So
I'll limit myself to just passthru variables.
This further improves the changes from bug 1224460 to e.g. handle variable
references mixed with text, and to avoid adding empty strings to the
resulting flags variables when the expansion leads to an empty string.
Pymake's clinetoargv is very specific to pymake's use case, yet has been abused
as a replacement for shlex because shlex doesn't handle things properly for our
use cases.
Using pymake's clinetoargv, however, has shortcomings, and we're better off
importing its code in mozbuild, simplifying it a little, and using that
instead.
Plus, less dependencies on pymake will help kill it for good some day.
FlatFormatter, JarFormatter and OmniJarFormatter all, in some way, deal
with different pieces of the package being handled differently.
Instead of each of them dealing with their different pieces in some subtly
different way, instead, introduce a new base package formatter class that
will handle it for all of them.
Use this new PiecemealFormatter for the FlatFormatter.
Only directories containing chrome manifests are given as base to formatters,
but there can still be files given outside the bases, like, on mac builds,
all files in Content/MacOS, or Content/Info.plist, whereas chrome manifests
are under Content/Resources.
There is a lot of repetition across its various tests, and we're going to add
some more in a subsequent change, so it is desirable to make it a less
repetitive task.
This function was found to be a little slow while profiling due to repeated calls to
mozpath.dirname. This patch speeds up the function replacing dirname with string manipulation
(these paths are already normalized), by caching results on the basis of directory,
and converting from iteration to recursion to increase use of the cache.
This commit speeds up the "install tests" step run as a part of the build and running
tests by ~10% on a fast linux laptop.
--HG--
extra : commitid : HdYkcXQ2ezQ
Support for displaying docstrings in `mach help` was added relatively
recently. `mach build` was never documented. Let's document it.
There are a gazillion things we could put in the documentation. For now,
mainly focus on targets.
--HG--
extra : commitid : FjtVDISK9Q5
extra : rebase_source : a69ba419e49ca0e4435e87597fdfe34623917a6c
extra : amend_source : 1161bf83569c82340ad1e4e4d21ba7f600753af1
When a make target is generated with FileAvoidWrite, this can cause targets to
get rebuilt perpetually when a prerequisite is updated, because FileAvoidWrite
will leave the target's mtime older than the prerequisite's when the target's
contents are unchanged. To avoid this issue, GENERATED_FILES is modified to
unconditionally update its target's mtime.
--HG--
extra : commitid : 4k5e5rKtPZ2
Bug 118468 landed an option for FileAvoidWrite to always write to an output
file, whether or not the contents would be changed. This was to address a
problem caused by not updating mtimes when building GENERATED_FILES, but
undoes the purpose of FileAvoidWrite and isn't really necessary.
This is addressed in a subsequent commit by unconditionally updating
mtimes when processing GENERATED_FILES.
--HG--
extra : commitid : AfOhgUstokq
Since MOZ_NATIVE_DEVICES builds against play-services-{basement,base,cast},
some ad-hoc de-duplication is necessary.
--HG--
extra : commitid : 2jNIgZpLUq2
extra : source : 0957d3435ac22765d7868cb3c7db1e0787836bc3
Calling CommonBackend.consume_object ensures that we process WebIDL and
IPDL files (and many other things) correctly. Calling
CommonBackend.consume_finished ensures that the CompileDB backend gets
to see the unified bindings and protocol files that we generate, and add
those files to the compilation database.
The only thing we need the obj for here is getting the objdir. Future
patches will just have an objdir when calling this function, and not a
proper mozbuild object. In light of these facts, let's change the
function to accept an objdir only, which will make those future patches
easier.
For GENERATED_FILES scripts that want to report dependencies, this
change makes it easy to use |preprocess|, rather than having to
construct and use |Preprocessor| manually.
In addition to their inputs declared in moz.build files, generated files
may also depend on other files, such as #includes in preprocessed files.
Let's provide a place for file_generate.py to write out those extra
dependencies.
Indicating a jar currently looks like the following in a jar manifest:
path/to/name.jar:
The `path/to` doesn't contain the implicit "chrome/" directory. This, in
turn, doesn't allow much flexibility to use the jar maker for what is not
necessarily under chrome/.
To use the jar maker to fill some chrome manifest for the default theme
extension, we currently use a hackish path to get to the right location,
and rely on the chrome.manifest file in the parent directory never to be
picked by the package manifest, which is a quite horrible way to do this,
but worked well enough for that specific use case.
With the need to handle system addons at the build system level, it
becomes necessary to come up with something less hackish.
What this change introduces is an additional syntax for the jar manifest,
in the following form:
[base/path] sub/path/to/name.jar:
Using this syntax, there is no implicit 'chrome' path. The `base/path` is
relative to the current DIST_SUBDIR, and the `sub/path` is relative to that
`base/path`. The distinction can be useful for build system backends.
The assumption that the "root" chrome.manifest is in the parent directory
of the implicit "chrome" directory dies, and the `base/path` is where the
root chrome.manifest is placed.
The bulk of this commit was generated with a script, executed at the top
level of a typical source code checkout. The only non-machine-generated
part was modifying MFBT's moz.build to reflect the new naming.
CLOSED TREE makes big refactorings like this a piece of cake.
# The main substitution.
find . -name '*.cpp' -o -name '*.cc' -o -name '*.h' -o -name '*.mm' -o -name '*.idl'| \
xargs perl -p -i -e '
s/nsRefPtr\.h/RefPtr\.h/g; # handle includes
s/nsRefPtr ?</RefPtr</g; # handle declarations and variables
'
# Handle a special friend declaration in gfx/layers/AtomicRefCountedWithFinalize.h.
perl -p -i -e 's/::nsRefPtr;/::RefPtr;/' gfx/layers/AtomicRefCountedWithFinalize.h
# Handle nsRefPtr.h itself, a couple places that define constructors
# from nsRefPtr, and code generators specially. We do this here, rather
# than indiscriminantly s/nsRefPtr/RefPtr/, because that would rename
# things like nsRefPtrHashtable.
perl -p -i -e 's/nsRefPtr/RefPtr/g' \
mfbt/nsRefPtr.h \
xpcom/glue/nsCOMPtr.h \
xpcom/base/OwningNonNull.h \
ipc/ipdl/ipdl/lower.py \
ipc/ipdl/ipdl/builtin.py \
dom/bindings/Codegen.py \
python/lldbutils/lldbutils/utils.py
# In our indiscriminate substitution above, we renamed
# nsRefPtrGetterAddRefs, the class behind getter_AddRefs. Fix that up.
find . -name '*.cpp' -o -name '*.h' -o -name '*.idl' | \
xargs perl -p -i -e 's/nsRefPtrGetterAddRefs/RefPtrGetterAddRefs/g'
if [ -d .git ]; then
git mv mfbt/nsRefPtr.h mfbt/RefPtr.h
else
hg mv mfbt/nsRefPtr.h mfbt/RefPtr.h
fi
--HG--
rename : mfbt/nsRefPtr.h => mfbt/RefPtr.h
The configure option has explicitly thrown an error for more than a year now,
and it happens that the remaining way to still forcefully use it has been
broken for more than 8 months.
DONTBUILD NPOTB
This downloads to a temporary file named uniquely but consistently
based on the URL, and then extracts a build ID using mozversion to use
as a human readable and sortable prefix. This approach can be re-used
by |mach artifact| based Desktop builds.
--HG--
extra : commitid : LxorDuq5D0t
extra : rebase_source : 2f280746f486b79dfe45ad928e4b618e0e12f1a0
The bulk of this commit was generated with a script, executed at the top
level of a typical source code checkout. The only non-machine-generated
part was modifying MFBT's moz.build to reflect the new naming.
# The main substitution.
find . -name '*.cpp' -o -name '*.cc' -o -name '*.h' -o -name '*.mm' -o -name '*.idl'| \
xargs perl -p -i -e '
s/nsRefPtr\.h/RefPtr\.h/g; # handle includes
s/nsRefPtr ?</RefPtr</g; # handle declarations and variables
'
# Handle a special friend declaration in gfx/layers/AtomicRefCountedWithFinalize.h.
perl -p -i -e 's/::nsRefPtr;/::RefPtr;/' gfx/layers/AtomicRefCountedWithFinalize.h
# Handle nsRefPtr.h itself, a couple places that define constructors
# from nsRefPtr, and code generators specially. We do this here, rather
# than indiscriminantly s/nsRefPtr/RefPtr/, because that would rename
# things like nsRefPtrHashtable.
perl -p -i -e 's/nsRefPtr/RefPtr/g' \
mfbt/nsRefPtr.h \
xpcom/glue/nsCOMPtr.h \
xpcom/base/OwningNonNull.h \
ipc/ipdl/ipdl/lower.py \
ipc/ipdl/ipdl/builtin.py \
dom/bindings/Codegen.py \
python/lldbutils/lldbutils/utils.py
# In our indiscriminate substitution above, we renamed
# nsRefPtrGetterAddRefs, the class behind getter_AddRefs. Fix that up.
find . -name '*.cpp' -o -name '*.h' -o -name '*.idl' | \
xargs perl -p -i -e 's/nsRefPtrGetterAddRefs/RefPtrGetterAddRefs/g'
if [ -d .git ]; then
git mv mfbt/nsRefPtr.h mfbt/RefPtr.h
else
hg mv mfbt/nsRefPtr.h mfbt/RefPtr.h
fi
--HG--
rename : mfbt/nsRefPtr.h => mfbt/RefPtr.h
