There are a lot of choices and moving pieces in this commit. I elected
to include the mechanics and the target use case in the same commit so
that readers can compare and contrast the implementation and final
expression in one review window.
- Initially, I wanted to make the {AB_CD} substitutions in
LOCALIZED_FILES and not in LOCALIZED_GENERATED_FILES. However, I ran
into conceptual blockers doing this. Fundamentally, LOCALIZED_FILES
is FINAL_TARGET_FILES, and my use case should _not_ be putting files
anywhere near dist/bin. In addition, LOCALIZED_FILES
(FINAL_TARGET_FILES) is handled using manifests, which would need to
grow locale-aware functionality to handle this. That's not desirable.
In addition, if we use manifests, then we lose the powerful locality
of |mach build mobile/android{/base}| re-generating changed
locale-dependent resources. This is similar to how the build system
plumbs dist/idl manifest processing throughout the build: we're
repairing local workflows after moving work into a global process.
For these reasons, this doesn't support {AB_CD} in LOCALIZED_FILES.
- There is even another layer of complexity! There are two axes
involved with these files: AB_CD controls localization and the Make
target controls destination. For the record, it is:
regular builds - AB_CD unset
multi-locale builds - AB_CD set
single-locale repacks - AB_CD set
For the record, the existing logic (before any changes) is:
regular builds - Make target is `libs` in mobile/android/base/locales
multi-locale builds - Make target is `chrome-%` in mobile/android/base/locales
single-locale repacks - Make target is `libs` in mobile/android/base/locales
This commit adds targets for both destinations, and uses Make
chrome-%:: and libs:: magic to control what is invoked in the various
situations. Tricky!
- I added MERGE_RELATIVE_FILES in order to be able to follow-up this
patch with more patches that will get rid of
m/a/base/locales/{moz.build,Makefile.in} altogether, and fold this work
into m/a/base. As it stands, we're already reaching from
m/a/base/locales all the way out to
mobile/locales/.../region.properties, so the existing code doesn't
follow the layout expected between mozilla-central and
l10n-central/$(AB_CD). But that'll impedance will get worse as we
improve the build system dependencies, not better, so we should grow
support for localized resources that aren't exactly as expected.
- I chose to follow Python's syntax for string substitutions. I
would have preferred to mark files that should be localized with a
leading '%'... but I took that for filesystem absolute paths in
moz.build files already. I also considered @AB_CD@ to echo the
preprocessor, but didn't want to open the door to an expecation that
_all_ preprocessor DEFINEs will work in the way {AB_CD} does.
- The generate_*py script changes required a bit of a hack to "turn
off" locale dependent resources. This would have been nicer if we had
marked localized resources with '%'... but we didn't. See the
--fallback flag. The real reason this is needed is that we're doing
work which is more like the work of compare-locales (merging
locale-dependent resources) at build-time rather than repack time. I
don't know why that's the case -- probably when we (I) implemented it,
compare-locales and the whole l10n process was entirely opaque. It's
not worth changing it now, so we use this --fallback flag approach.
- I didn't get to tup support. This should gently fail without
breaking tup builds: any {AB_CD} substitutions just won't be
expanded. I haven't a clue how this should work in tup in the future
(or, more generally, how to make any sense of repacks without
declaring the full set of expected locales at configure time.)
- strings.xml can't be a LOCALIZED_PP_FILES, since we need to
customize the output location based on AB_rCD, and since we need a
little more flexibility than PP_FILES gives for our inputs.
MozReview-Commit-ID: MyfIkNSEzt
--HG--
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/en-US/localized-input => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/en-US/localized-input
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/foo-data => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/foo-data
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/generate-foo.py => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/generate-foo.py
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/en-US/localized-input => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/inner/locales/en-US/localized-input
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/moz.build => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/moz.build
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/non-localized-input => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/non-localized-input
extra : rebase_source : 816b6f220758f2bb3bdd3ec81a2cb02269c6de5b
Without linking support in the tup backend, we don't produce a js
binary, so we can't copy it via OBJDIR_FILES yet.
MozReview-Commit-ID: AxqhHi84HIg
--HG--
extra : rebase_source : 1011bc6296be81646004a6bd6572193c34f367d8
Starting with Rust 1.24, the default codegen-units limit is 16,
with jobserver control to avoid overprovisioning. Remove our
previous fixed limit of 4 threads for debug builds.
For release, retain codegen-units=1 to make sure we get the
most complete optimization results.
Thanks to Simon Sapin for the suggestion.
MozReview-Commit-ID: FmYF4DcmBvt
--HG--
extra : rebase_source : 307ad8fad2874636adb3ce95a5cd47339e83f40c
The Proguard dependency is now managed by Gradle.
MozReview-Commit-ID: EOvKSE5z28P
--HG--
extra : rebase_source : 760b117f500cc639cc8c24e9c02933990f358dd7
The moz.build Java JAR definitions are, of course, broken, but they
will be removed soon enough.
MozReview-Commit-ID: KIxqLDwd9I7
--HG--
extra : rebase_source : 8312b3f125793f73d3e835d1c0a5c7cabd4ebc0c
I choose to clean a bunch of ANDROID_* moz.build cruft here, too,
since it's just passing dependencies between moz.build and
Makefile.in. The replacement for all of this is to just use
GENERATED_FILES in moz.build, but it'll still take some work to get to
that. (Why does this stuff exist? GENERATED_FILES didn't exist and
was resisted when I built this stuff.)
MozReview-Commit-ID: D3GJqJNL0Ih
--HG--
extra : rebase_source : 07351f9d3702cfc42c58bd317885d07882c45c3a
This is the easy stuff -- everything but mobile/android/base/Makefile.in.
MozReview-Commit-ID: 5x2z97AHUrR
--HG--
extra : rebase_source : 531fd41d367cad071b209b85ca5b5602fd7cbf7b
The last APK produced using the ANDROID_APK_* moz.build/Makefile.in
mechanism was Robocop, so we can get rid of these now.
MozReview-Commit-ID: 9b08ZvvOAoC
--HG--
extra : rebase_source : ac4fea057bf6e731b0f26a1b6902f17a7362076d
The invalid variable test for #if{,n}def was only checking that the
first character in the variable was alphanumeric or underscore, not
the other characters.
More generally, preprocessor instructions were also cut out such that
whitespaces before and after arguments were part of the arguments.
Subtly, some legitimate strings end with what, in ISO-8859-1, is
considered as whitespaces, and because the preprocessor largely works
on byte strings (str), and because the regexps are using re.U, those
characters (e.g. 0xa0) that can legitimately appear in byte strings of
UTF-8 encoding, are treated as whitespaces. So we remove the re.U from
the instruction regexp, so that only plain ascii whitespaces only are
stripped out.
There's one place in layout/tools/reftest/manifest.jsm that was using
a broken pattern, making the test never true, which, once fixed, unveils
broken tests, so the branch that was never used is removed.
--HG--
extra : rebase_source : b695dec025c55aee0e50f2a0047278fe9c849c9e
Depending on the compiler you use when --enable-stdcxx-compat, the
compiler can know about different libstdc++.so libraries that are not
suitable for your target. This will manifest as an assertion in the
current libstdcxx.py file. And then, when you change the assertion to
actually print out useful information, you will see things like:
/bin/ld: skipping incompatible /builds/worker/workspace/build/src/clang/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.4/../../../libstdc++.so when searching for -lstdc++
/bin/ld: skipping incompatible /builds/worker/workspace/build/src/clang/bin/../lib/libstdc++.so when searching for -lstdc++
which libstdcxx.py misinterprets as candidates for libstdc++.so.
This patch attempts to remedy both situations by providing a more
informative error message when things go sideways and also filtering out
error messages from the linker. You could argue that perhaps
--enable-stdcxx-compat shouldn't be getting set for such builds, but
this change seems reasonable enough on its own.
The invalid variable test for #if{,n}def was only checking that the
first character in the variable was alphanumeric or underscore, not
the other characters.
More generally, preprocessor instructions were also cut out such that
whitespaces before and after arguments were part of the arguments.
There's one place in layout/tools/reftest/manifest.jsm that was using
a broken pattern, making the test never true, which, once fixed, unveils
broken tests, so the branch that was never used is removed.
--HG--
extra : rebase_source : d1fe8a299203a29c0906ff99054c326acd135000
Right now, the "restart flow" that combines |mach watch| with the
Quick-Restart Firefox for Desktop shortcut key is frustrated by
inconsistencies writing the .purgecaches sentinels for the
application. This commit uses recent work from
https://bugzilla.mozilla.org/show_bug.cgi?id=1368699 to write the
sentinels each time |mach watch| updates the object directory.
MozReview-Commit-ID: 62Aa85oT1SE
--HG--
extra : rebase_source : 746bbe5c6f1555e8b729cbbbc1f8ca57110ae9ba
IntelliJ should still work, but we're committed to Android Studio at
this point.
MozReview-Commit-ID: 3BaXB4dh4vA
--HG--
extra : rebase_source : 0bf39a7d8daddc9a5c74182cf266f5d01d17acc8
extra : source : 901b304603d9d4816856d560c61387501efceadc
This adds a new `post_build` step to each `BuildBackend`
implementation, and uses it to write .purgecaches after every |mach
build| invocation -- including after |mach build TARGET| invocations.
This approach should be more robust than the existing recursive-Make
based solution, which seems to not write the .purgecaches files in
some situations.
In addition, the recursive-Make solution does not generalize to other
backends, in particular Tup. It is possible that the Tup backend will
handle writing the .purgecaches sentinel as part of its regular build
process, but discussions with mshal suggest that there's no convenient
way for Tup to write .purgecaches only when something *changes* during
the build. That is, Tup can achieve the behaviour implemented by this
patch, but it's not easier to do better by not writing .purgecaches
when the caches do not in fact need to be purged.
I elected to bake in the special knowledge of
--enable-application=browser and macOS here since this whole process
is special. If we need to generalize, we could add a moz.configure
option specifying the purgecaches directories, but it doesn't seem
worth it right now.
The ideal approach would be to determine FINAL_TARGET from the
application directory, but that is determined by DIST_SUBDIR. In
addition, it's not clear how to present that information to the
post-build step in a build-backend agnostic manner.
This will require tweaking as we migrate the macOS bundle handling to
moz.build, especially in browser/app. See
https://bugzilla.mozilla.org/show_bug.cgi?id=1223748, which could
improve this significantly.
MozReview-Commit-ID: 63KZy18D23i
--HG--
extra : rebase_source : e973d065cd91e965f4103ed2732858e2e7a9c546
This adds a new `post_build` step to each `BuildBackend`
implementation, and uses it to write .purgecaches after every |mach
build| invocation -- including after |mach build TARGET| invocations.
This approach should be more robust than the existing recursive-Make
based solution, which seems to not write the .purgecaches files in
some situations.
In addition, the recursive-Make solution does not generalize to other
backends, in particular Tup. It is possible that the Tup backend will
handle writing the .purgecaches sentinel as part of its regular build
process, but discussions with mshal suggest that there's no convenient
way for Tup to write .purgecaches only when something *changes* during
the build. That is, Tup can achieve the behaviour implemented by this
patch, but it's not easier to do better by not writing .purgecaches
when the caches do not in fact need to be purged.
I elected to bake in the special knowledge of
--enable-application=browser and macOS here since this whole process
is special. If we need to generalize, we could add a moz.configure
option specifying the purgecaches directories, but it doesn't seem
worth it right now.
The ideal approach would be to determine FINAL_TARGET from the
application directory, but that is determined by DIST_SUBDIR. In
addition, it's not clear how to present that information to the
post-build step in a build-backend agnostic manner.
This will require tweaking as we migrate the macOS bundle handling to
moz.build, especially in browser/app. See
https://bugzilla.mozilla.org/show_bug.cgi?id=1223748, which could
improve this significantly.
MozReview-Commit-ID: 63KZy18D23i
--HG--
extra : rebase_source : c5f75192f1b77f08611662f51caa92dcb1ca802f
Right now, we only expect classes.dex, and even --with-gradle we copy
it out of $topobjdir/mobile/android/base. This commit changes that
for --with-gradle: we only take classes.dex from the given .ap_ file,
and we also handle multiple classesN.dex files (for future multi-dex
support). The moz.build system stays the same.
This avoids an issue with newer Android-Gradle plugins, where the
classes.dex produced could be either in dex/ or in dexMerger/,
depending on whether any external libraries needed merging. By
extracting classes.dex from the .ap_ file, we don't need to know what
Gradle build steps actually occur.
The classes.dex in the package-manifest.in has been irrelevant since
Bug 1260241.
MozReview-Commit-ID: FozKwjTcMzU
--HG--
extra : rebase_source : 62b18c7ffe596be73cec4c9565333eac222b018e
There's a bug in InstallManifest::add_entries_from: some of the
manifest entries bake their destination into both the manifest key and
the manifest value, and add_entries_from with a non-empty "base"
parameter to prepend to the destination only updates the manifest key
and not the value.
This bug causes |mach watch| to fail to _read_ the unified manifest
that aggregates all of the build manifests relevant to |mach watch|
that |mach build-backend -b FasterMake| successfully _wrote_, because
the manifest keys are validated against the manifest values written to
disk.
I see no way to address this other than to manually reach into the
manifest values and patch the internal destination parameter, which
this patch does.
MozReview-Commit-ID: FVyRB41NnHa
--HG--
extra : rebase_source : 23eb18ddc0452955539ce2e7a6d7bbfd083c940c
