JAR_MANIFESTS affects the libs tiers through config/rules.mk rules.
While we could move the rules in the backend, they are too complex to
just do that now.
GENERATED_FILES impacts the export tier through the config/rules.mk
definitions, now moved to the backend itself, so that everything is
close to each other.
Those are non-passthru variables with the same property as
ANDROID_GENERATED_RESFILES, ANDROID_APK_NAME and ANDROID_APK_PACKAGE, in
that they don't affect tiers through the backend code itself, but from
the Makefile.in along the moz.build they are defined in.
The are passthru variables that don't actually affect any tier per the
backend itself. They do affect the `export` tier by way of the Makefile.in
along the moz.build defining them, and the existence of those
Makefile.in already guarantees those directories not to be skipped for
`export`.
This initiates a move off affected_tiers in VARIABLES definition to
explicit in-backend handling, which will hopfully make things clearer.
HAS_MISC_RULE is currently used to opt-in to the misc tier in a few
directories with a misc:: rule. It is, in fact, mostly used for custom
xpi creation, which will be separately addressed in bug 1240676, so it
will eventually go away entirely, but in the meantime, we send it as a
throwaway passthru.
Historically, all tiers were handled as opt-out (may_skip), until we
added the first opt-in tier (no_skip). It doesn't make much sense to
differentiate them anymore, so handle them all as opt-in.
When adding a backend, we currently have to add them in three different
places so that they appear in the right places.
Instead, keep the list in a single place.
Currently |./mach artifact| installs an old version of the taskcluster client,
which installs an old version of requests that's incompatible with commonly
installed python versions. This bumps to a version of taskcluster client that
accepts and requests version < 3, so we pick up the in-tree version instead of
installing 2.4.3.
--HG--
extra : rebase_source : 7b4f450c0492fd3c840e4f0a0cce8b42d120df9f
generate_symbols_file only supports the global defines, and completely
ignores DEFINES from the same moz.build the SYMBOLS_FILE is defined.
This fixes this misfeature.
A recent change regressed this behavior -- while an artifact build runs, it
doesn't load certain "about:" pages due to missing libraries in subdirectories
of dist/bin.
--HG--
extra : commitid : HOPgt9kMKoV
AFAICT, we don't actually access web-platform test files from the
objdir for anything except test packaging. And we already have
a mechanism for creating test archives from files directly in the
source directory. So, let's stop copying them to the objdir and
package them directly from the source directory!
The _tests install manifest reports the following change:
before: 41,977 files installed
after: 24,537 files installed
delta: -17,440 files
We still copy some WPT files to the objdir. We might be able to
eliminate these as well. However, since there are only ~200 files,
I'm not too concerned.
I manually compared the resulting web-platform zip archives from before
and after. No files were removed from the archive. However, the new
archive does gain several hundred empty directories with .gitkeep
files. This feels weird, but it shouldn't break anything (I would
think). I'm inclined to leave them for now. I'll file a follow-up
bug to deal with them (preferably by removing them from version
control).
--HG--
extra : rebase_source : d350f8fc223982c8a11a9bf542411e5ec5e44244
The faster make backend cannot support such things, so it's better to
avoid unsupported things to slip in because it happens that doesn't
break what automation runs.
Right now, each incremental build produces a new set of GeckoView
library files, but the previous files don't get cleaned up, and you end
up with a bunch of old libraries in your objdir. This patch cleans up
the old files before producing new ones.
See https://trac.macports.org/ticket/48807. Without it, reading
input can break terminals.
--HG--
extra : rebase_source : 7eab1e60d6a6279fdaf25a4789598cf96e2b8d6c
Also fix that the default merge dir in the mach command creates a directory
that's the merge make target, and thus keeps that make target from actually
running.
--HG--
extra : transplant_source : R%FE%7Eo%CDS%D1%B4%EA%DB%D0%19q%F92%A1V%91%10%5C
Collecting the list of object files compiled, while not ideal, will give us
some indication of how much work was involved in the build. This will help
with analyzing the data.
--HG--
extra : rebase_source : e9861ed5c0766e3ee8038dbec0b9267022c523eb
This adds metadata about the local system to resource_usage.json that will
be useful for analysis when we start collecting these results. Some of it
is redundant with data collected for individual tiers, but having it
available at a top level should make analysis easier.
--HG--
extra : rebase_source : 43b2f6335a4de9c7843f75a3461d2871a97b7752
This moves monitor start/end recording so it also occurs for "what"
builds rather than only for full builds.
--HG--
extra : rebase_source : 833c7311ca5ecfe6935e0a160da695acaecbdeeb
Pushing on a CLOSED TREE after network errors.
--HG--
extra : commitid : 40hmCLC8zZJ
extra : rebase_source : 44a623fdf7cb97bfaca76617062d57c6b50fd970
extra : amend_source : 710db8302b5bf88d6bf293d71f3cc001187b6015
The result of running this backend is a json file containing an array with 3 elements:
a map from resource:// and chrome:// url prefixes to corresponding objdir directories,
a map of overrides, and a map from objdir files to srcdir files (with a flag for whether
the file was preprocessed) for js file in FinalTargetFiles. A subsequent commit implements
the procedure to find an replace based on the url mapping, subsistute based on overrides,
and find sourcedir files based on objdir files.
--HG--
extra : commitid : Di6Ro5c9sTK
This mounts the downloaded DMG and copies a subset of the libraries
into the correct places in the processed archive. They'll be
installed, with paths, into dist/bin from there.
--HG--
extra : commitid : 9BXzXzIlh3I
extra : rebase_source : 29ddbe8348f27fff5205da6c40839a04d950fe0a
For Android, this just copies .so files from the APK assets/ and libs/
directories into the root, which ends up in dist/bin.
I include .ini files as a partial replacement for having the build ID
in the downloaded file name, and then don't put the .ini files in
dist/bin.
--HG--
extra : commitid : 9uLXEzBNBdQ
extra : rebase_source : 53ba0e8a7740c2a6eb48e2772f7e12ff9157d21a
Sadly, it's slow to extract the build ID from Mac OS X DMG artifacts.
It's better to sacrifice human-legible names in order to know the
final name for an artifact quickly.
--HG--
extra : commitid : I99dtzv1O2N
extra : rebase_source : c164be1314b3132fba2e1fc86cda65b12b4baa1f
It turns out to be much easier to hook |mach artifact install| into
config.status and |mach build| than to hook into client.mk.
The additional virtualenv package avoids an import error when running
|mach artifact install|.
--HG--
extra : commitid : EnfWU0uyRfQ
extra : rebase_source : f7d11fc4c542f9798712c013c4319d92d40c28e5
StrictVersion is strict about version strings, insisting on whatever
convention Python uses. LooseVersion is not as strict but is strict
enough for our use cases.
DONTBUILD (NPOTB)
--HG--
extra : commitid : 17lNEAJhaV0
extra : rebase_source : 0a0cefa47b4558401cb85c6e9b237c0d6cf0e7fb
extra : amend_source : c7360d1a2f934338ec04d5f384d4530e3e9ebbc5
3.5.2 is what is listed in `mach mercurial-setup`. These should match.
Add a comment to each file saying to change both.
--HG--
extra : commitid : FebjTovmqGk
extra : rebase_source : 50490c1896a4c402f27cf4154b155932614da558
extra : amend_source : 73ae0ddc9f2770351d2ee2aaf5121656fb7e5750
Limit ourselves to include paths for now, because there are tricky things
involved in making this globally.
While here, use shell_quote instead of manual quoting for those paths.
With all include flags now using absolute paths, there is no need to try
to post-process them when getting them for CompileDB and codecomplete.
As a matter of fact, doing so fixes the flags in media/gmp-clearkey/0.1,
since they use literal "-include stdio.h", which was wrongly transformed
into "-include $objdir/media/gmp-clearkey/0.1/stdio.h".
MOZ_DEBUG_DEFINES are essentially defines used everywhere. So treat them as
feeding the initial value for DEFINES in each moz.build sandbox. This allows
the kind overrides that was done in the past by resetting MOZ_DEBUG_DEFINES
in Makefiles.
Currently, one needs to define DEFFILE or LD_VERSION_SCRIPT appropriately,
and somehow deal with the fact that their input format is different, which
currently relies on manual invocations of the convert_def_file script, with
awkward aggregations.
This simplifies the problem by using a simple list of symbols, with
preprocessing, allowing #includes.
We want to move it to CommonBackend, so it's better to make it more
independent, which the Defines instances now attached to ContextDerived
instances allow.
Like with ChromeManifestEntries, reloop in consume_object, with the double
goal of allowing to reuse the jar manifest handler code in other backends
and avoid code duplication in the FasterMake backend itself when support
for e.g. GeneratedFiles is added.
Instead of filling the install manifests accordingly, reloop in
consume_object, so that the jar manifest handler code can eventually
be reused in other backends.
Currently, only css files added through jar manifests are treated this way.
There is really no reason for the discrepancy, but there are actually no css
files added directly through moz.build, so this was never a problem.
On the other hand, it makes things simpler in a world where jar manifests are
treated as if they were entirely described in moz.build (which is where the
FasterMake backend is heading).
Again, this is not strictly necessary but allows to confirm idempotence of
further changes. And it has the nice side effect of making chrome manifest
files more consistent.
Using TEST_DIRS is nothing more than a shortcut for
if CONFIG['ENABLE_TESTS']:
DIRS += [...]
As such, we might as well remove it being a separate variable, and use some
Context magic to just fill DIRS when ENABLE_TESTS is set.
The security/manager/ssl/tests/unit/moz.build change ensures that the order
of DIRS before the change is kept, not because it matters, but because it
allows to confirm that nothing else is modified by this change.
Bug 1191230 added override lines with # characters to chrome manifests
for Windows.
So far, chrome manifests were handled with buildlist.py like in the
RecursiveMake backend, fed with Make variables. Without proper quoting,
those Make variables are just truncated by Make on the first # character,
and this results in mach build faster failing because of that.
However, the reason why chrome manifests were handled with buildlist.py
originally is that not all chrome manifest entries were known to the
FasterMake backend, but they now all are.
So instead of relying on Make variables and buildlist.py, we can now
rely on the newly added install manifests feature allowing to create files
with a given content.
From the backend perspective, CONFIGURE_DEFINE_FILES is the same as
GENERATED_FILES because in both cases a GeneratedFile object is emitted, but
from the perspective of some checks in the emitter, they aren't the same,
and that causes errors when adding a CONFIGURE_DEFINE_FILES to e.g. EXPORTS.
Also removes related unused variables in mach_commands.py.
--HG--
extra : commitid : IiDVMuEZtA5
extra : rebase_source : 575a51dd0ad5450323b4da5f441f8e5d721e41d6
From the backend perspective, CONFIGURE_DEFINE_FILES is the same as
GENERATED_FILES because in both cases a GeneratedFile object is emitted, but
from the perspective of some checks in the emitter, they aren't the same,
and that causes errors when adding a CONFIGURE_DEFINE_FILES to e.g. EXPORTS.
Running old extensions with newer versions of Mercurial may crash `hg`
due to the old extension accessing something or doing something that has
been changed in the new release.
To minimize the risk of this happening, we disable common 3rd party
extensions when cloning or pulling as part of `mach mercurial-setup`. We
don't want to disable everything because some extensions (like
remotenames) provide features the user may want enabled as part of the
clone/update. This leaves the door open for more failures. Hopefully
this approach is sufficient. We can always revisit later.
--HG--
extra : rebase_source : 92e7d8fe227f29fc64c0f69021bd731ba762faf3
In order to use StrictOrderingOnAppendListWithFlags instances in
mozbuild template functions, we need += to work correctly. This patch
implements extend and the associated functions (including +=),
disallowing some behaviour where convenient.
There's a subtle point hidden in the isinstance() tests: before this
patch, it was not easy to compare two
StrictOrderingOnAppendListWithFlags instances to see if they had the
*same* set of flags. That was because two instances may not have the
same class, and would only share the common
StrictOrderingOnAppendList, which isn't enough to infer the presence
of flags. To be slightly more clear, concrete instances will have
class StrictOrderingOnAppendListWithFlagsSpecialization (although
there are still multiple instances of that class) and all extend from
the unique class StrictOrderingOnAppendListWithFlags.
--HG--
extra : commitid : AMVDYt8khR
extra : rebase_source : 1ce0698691fc03fbdf6a976e92017c1d60bad15d
extra : histedit_source : 4812a565179fb4fac2e4b5cd89c4efe74e794dfa
DONTBUILD NPOTB
The top source directory configuration requires
mobile/android/gradle/m2repo/**, so it stays. There's no value
changing the location; it contains an Android-specific Gradle plugin.
We note the removal of |mach gradle-install| and point to the new
documentation.
--HG--
extra : commitid : 9Nhz2dnBIgY
extra : rebase_source : 32a2b8a92d57f963feac2bae28fed5a9f1b26f93
extra : amend_source : bf53a0b2d3d4ac0618bc82fe79914bdeaf1c1e0a
Both these files, are, after all, define files, like other CONFIGURE_DEFINE_FILES.
They only happen to have a special requirement for an expansion for all defines,
which doesn't need to happen through traditional preprocessing.
This change adds consistency in how configure-related headers are being handled.
This is needed to support hgwatchman.
--HG--
extra : commitid : 8D2A8YPNimB
extra : rebase_source : 7d5932aa049dfb352b93a87c2c8087dd7b324aab
extra : histedit_source : 9863189f265eca9e0b9363e13c59a7d55f5c633d
This change allows specifying objdir-relative paths in EXPORTS to enable
exporting entries from GENERATED_FILES. Objdir paths in EXPORTS that are
not in GENERATED_FILES will raise an exception.
Example:
```
EXPORTS += ['!g.h', 'f.h']
GENERATED_FILES += ['g.h']
```
Given the implementation, this should also work for FINAL_TARGET_FILES,
FINAL_TARGET_PP_FILES, and TESTING_FILES, but those are not well-tested.
This patch also renames the install manifest for '_tests' to match the
directory name for convenience in some code I refactored.
--HG--
extra : commitid : CwayzXtxv1O
extra : rebase_source : 5fb6f461fc740da9bce14bbdbfabdfe618af8803
Future improvements to process_install_manifest's --track option will require
adding data in the tracking dump that uses an install manifest form, and I don't
want e.g. switching branches or bisection to require to clobber in order to do the
right thing, so this change future-proofs the install manifest reader.
There are currently two operating modes for process_install_manifest:
- default, which removes any file in the destination directory that is not
in the install manifest
- --no-remove, which doesn't do the above.
While install manifests also have the ability to deal with files that may
be left in the destination directory some other way, that requires knowing the
list of those files in advance, which is not always possible.
For instance:
- with the FasterMake build backend, install manifests are split such that
there is one manifest per application of addon directory (to allow more
parallelism), which means there is one for dist/bin and one for several
of its sub-directories.
- With --disable-compile-environment combined with artefacts, the backends
are not aware of e.g. all the libraries and executables that end up in
dist/bin.
If we want to properly remove files when they are removed from moz.build
or jar.mn, we can't use --no-remove, but the alternative would remove those
files
So add an option that keeps a list of all the files that were installed as
part of processing the given install manifest(s). That information is simply
a dump of the install manifest, which, while it contains more information
than currently required, will allow to do smarter things in the future.
The default behavior for a FileCopier's copy is to remove all the files and
directories in the destination that aren't in its registry.
The remove_unaccounted argument can be passed as False to disable this
behavior.
This change adds another possibility, where remove_unaccounted may be a
FileRegistry, in which case only the files in that registry are removed.
This allows to e.g. only remove files that were copied from a previous
FileCopier.copy, leaving aside files that were in the destination for some
other reason.
Future improvements to process_install_manifest's --track option will require
adding data in the tracking dump that uses an install manifest form, and I don't
want e.g. switching branches or bisection to require to clobber in order to do the
right thing, so this change future-proofs the install manifest reader.
There are currently two operating modes for process_install_manifest:
- default, which removes any file in the destination directory that is not
in the install manifest
- --no-remove, which doesn't do the above.
While install manifests also have the ability to deal with files that may
be left in the destination directory some other way, that requires knowing the
list of those files in advance, which is not always possible.
For instance:
- with the FasterMake build backend, install manifests are split such that
there is one manifest per application of addon directory (to allow more
parallelism), which means there is one for dist/bin and one for several
of its sub-directories.
- With --disable-compile-environment combined with artefacts, the backends
are not aware of e.g. all the libraries and executables that end up in
dist/bin.
If we want to properly remove files when they are removed from moz.build
or jar.mn, we can't use --no-remove, but the alternative would remove those
files
So add an option that keeps a list of all the files that were installed as
part of processing the given install manifest(s). That information is simply
a dump of the install manifest, which, while it contains more information
than currently required, will allow to do smarter things in the future.
The default behavior for a FileCopier's copy is to remove all the files and
directories in the destination that aren't in its registry.
The remove_unaccounted argument can be passed as False to disable this
behavior.
This change adds another possibility, where remove_unaccounted may be a
FileRegistry, in which case only the files in that registry are removed.
This allows to e.g. only remove files that were copied from a previous
FileCopier.copy, leaving aside files that were in the destination for some
other reason.
The only use of BRANDING_FILES[...].source is in xulrunner/app/moz.build, for
the app.ico file.
This file has not been useful since the removal of the xpinstall-based
installer in bug 344236... 9 years ago.
Currently mach treats the first argument to eslint as the path and moves it to
the end of the arguments but this breaks usage like "mach eslint -f json browser".
It used to be necessary to change to the directory you wanted to lint but now
the .eslintignore is at the top level we just run from the top level. This means
the path argument doesn't need to be special anymore.
--HG--
extra : commitid : 5ozct0pVSC4
extra : rebase_source : 22132a240d8e6f4d099dbcdeb793958d7173e154
extra : amend_source : 2b9931b4283e1c84f699027e13eccc33fcdec978
The current implementation of HierarchicalStringList allows the following:
FOO.bar = [
'foo',
'bar',
]
while
FOO.bar += [
'foo',
'bar',
]
would be invalid because of the StrictOrderingOnAppendList enforcement.
It also allows to overwrite the entire list with a subsequent
FOO.bar = [
'baz',
]
while we've explicitly forbidden such things for every other list.
While in the vicinity, fix HierarchicalStringList._get_export_variable to not
call the HierarchicalStringList constructor uselessly.
This will allow a new kind of special variable where it is possible to do
FOO += ['bar']
All the current special variables are either strings (for which __setitem__ would
be called with a different string object), or a read-only dict (which doesn't
allow modifications).
We have many unit tests in the tree for some small parts of the build system
pipeline, but we don't have anything that resembles an end to end test, and we
kind of rely on the resulting Firefox not being broken by our changes.
With the Faster make backend growing, I want to ensure it produces the same
thing as the recursive make backend, at least for the parts it supports.
This adds some kind of test that allows to check that.
The test I'm about to add doesn't have XPIDL files, and that currently avoids
the FasterMake backend to run properly. Also, in the future, when the FasterMake
grows the ability to build C++ files, it should be possible to build Spidermonkey
with the FasterMake backend, but it doesn't have XPIDL files either.
This makes it clearer that really it's the same thing as FINAL_TARGET,
with preprocessing.
We still keep DIST_FILES in backend.mk because it's shorter and doesn't
really matter.
This new ChromeManifestEntry object type is generic and can hold any kind of
chrome manifest entry, but we currently only emit them for binary components.
References to sub-directory manifests is left to the backend, for now, until
all manifest entries are emitted by the frontend.
Ideally, we should properly make and shell quote everything we print out
in makefiles, but that's a can of worms I don't want to open just yet. So
I'll limit myself to just passthru variables.
This further improves the changes from bug 1224460 to e.g. handle variable
references mixed with text, and to avoid adding empty strings to the
resulting flags variables when the expansion leads to an empty string.
Pymake's clinetoargv is very specific to pymake's use case, yet has been abused
as a replacement for shlex because shlex doesn't handle things properly for our
use cases.
Using pymake's clinetoargv, however, has shortcomings, and we're better off
importing its code in mozbuild, simplifying it a little, and using that
instead.
Plus, less dependencies on pymake will help kill it for good some day.
FlatFormatter, JarFormatter and OmniJarFormatter all, in some way, deal
with different pieces of the package being handled differently.
Instead of each of them dealing with their different pieces in some subtly
different way, instead, introduce a new base package formatter class that
will handle it for all of them.
Use this new PiecemealFormatter for the FlatFormatter.
Only directories containing chrome manifests are given as base to formatters,
but there can still be files given outside the bases, like, on mac builds,
all files in Content/MacOS, or Content/Info.plist, whereas chrome manifests
are under Content/Resources.
There is a lot of repetition across its various tests, and we're going to add
some more in a subsequent change, so it is desirable to make it a less
repetitive task.
This function was found to be a little slow while profiling due to repeated calls to
mozpath.dirname. This patch speeds up the function replacing dirname with string manipulation
(these paths are already normalized), by caching results on the basis of directory,
and converting from iteration to recursion to increase use of the cache.
This commit speeds up the "install tests" step run as a part of the build and running
tests by ~10% on a fast linux laptop.
--HG--
extra : commitid : HdYkcXQ2ezQ
Support for displaying docstrings in `mach help` was added relatively
recently. `mach build` was never documented. Let's document it.
There are a gazillion things we could put in the documentation. For now,
mainly focus on targets.
--HG--
extra : commitid : FjtVDISK9Q5
extra : rebase_source : a69ba419e49ca0e4435e87597fdfe34623917a6c
extra : amend_source : 1161bf83569c82340ad1e4e4d21ba7f600753af1
When a make target is generated with FileAvoidWrite, this can cause targets to
get rebuilt perpetually when a prerequisite is updated, because FileAvoidWrite
will leave the target's mtime older than the prerequisite's when the target's
contents are unchanged. To avoid this issue, GENERATED_FILES is modified to
unconditionally update its target's mtime.
--HG--
extra : commitid : 4k5e5rKtPZ2
Bug 118468 landed an option for FileAvoidWrite to always write to an output
file, whether or not the contents would be changed. This was to address a
problem caused by not updating mtimes when building GENERATED_FILES, but
undoes the purpose of FileAvoidWrite and isn't really necessary.
This is addressed in a subsequent commit by unconditionally updating
mtimes when processing GENERATED_FILES.
--HG--
extra : commitid : AfOhgUstokq
Since MOZ_NATIVE_DEVICES builds against play-services-{basement,base,cast},
some ad-hoc de-duplication is necessary.
--HG--
extra : commitid : 2jNIgZpLUq2
extra : source : 0957d3435ac22765d7868cb3c7db1e0787836bc3
Calling CommonBackend.consume_object ensures that we process WebIDL and
IPDL files (and many other things) correctly. Calling
CommonBackend.consume_finished ensures that the CompileDB backend gets
to see the unified bindings and protocol files that we generate, and add
those files to the compilation database.
