Historically we built all our binaries in directories in the objdir, then
symlinked them into dist/bin. Some binaries needed to be copied instead
so that certain relative path lookups work properly, so we resorted to
sprinkling `NSDISTMODE=copy` around Makefiles.
This change makes it so we build PROGRAMs (not any other sort of targets)
directly in dist/bin instead. We could do the same for our other targets
with a little more work.
There were several places in the tree that were copying built binaries to
some other place and needed fixup to match the new location of binaries.
On Windows pdb files are left in the objdir where the program was
originally linked. symbolstore.py needs to locate the pdb file both to
determine whether it should dump symbols for a binary and also to copy
the pdb file into the symbol package. We fix this by simply looking for
the pdb file in the current working directory if it isn't present next
to the binary, which matches how we invoke symbolstore.py.
MozReview-Commit-ID: 8TOD1uTXD5e
There a few pieces needed here to properly handle KeyboardInterrupts.
1. All in-progress work needs to abort. Ideally the underlying linters will be
able to catch KeyboardInterrupt, and return partial results (like the flake8
linter does). Linters may alternatively allow the KeyboardInterrupt to
propagate up. Mozlint will catch and handle this appropriately, though any
results found will be lost. The only change to this behaviour was fixing a bug
in the flake8 linter.
2. Any unstarted jobs need to be canceled. In concurrent.futures, there are two
different queues. First, jobs are placed on the work queue, which is just a list
maintained by the parent process. As workers become available, jobs are moved
off the work queue, and onto the call queue (which is a multiprocessing.Queue).
Jobs that live on the work queue can be canceled with 'future.cancel()', whereas
jobs that live on the call queue cannot. The number of extra jobs that are stored
on the call queue is determined by this variable:
https://hg.mozilla.org/mozilla-central/file/deb7714a7bcd/third_party/python/futures/concurrent/futures/process.py#l86
In this patch, the parent process' sigint handler (which will be called on Ctrl-C)
is responsible for canceling all the jobs on the work queue. For the jobs on the
call queue, the best we can do is set a global variable that tells workers to
abort early.
3. Idle workers should exit gracefully. When there are no more jobs left, workers
will block on the call queue (either waiting for more jobs, or waiting for the
executor to send the shutdown signal). If a KeyboardInterrupt is received while a
worker is blocking, it isn't possible to intercept that anywhere (due to quirks
of how concurrent.futures is implemented). The InterruptableQueue class was
created to solve this problem. It will return None instead of propagating
KeyboardInterrupt. A None value will wake the worker up and tell it to gracefully
shutdown. This way, we avoid cryptic tracebacks in the output.
With all of these various pieces solved, pressing Ctrl-C appears to always exit
gracefully, sometimes even printing partial results.
MozReview-Commit-ID: 36Pe3bbUKmk
--HG--
extra : rebase_source : d4c312ee5cc3679eeee1407c5521aed679f84ad4
extra : source : a93a00141bf62f6bc9e30934c0e56f6b2e434bf0
This commit doesn't change any behaviour, just attempts to make this a little
more readable. The workers will call '_collect_results' for each WorkItem they
process (either because it is finished or because it was canceled).
This also differentiates between setup failures and run failures.
MozReview-Commit-ID: 36Pe3bbUKmk
--HG--
extra : rebase_source : 873167512b745ccdc52de7e7f1ecf66b094e063d
This argument does nothing. While that's arguably a bug, I have
no desire to fix it. So remove dead code.
MozReview-Commit-ID: 9tToF66I7HE
--HG--
extra : rebase_source : 2ea86681a102d3a82fc547f52e473f4a46a60467
I was too lazy to find the commit that orphaned this. But it is most
definitely not referenced in the code base.
MozReview-Commit-ID: 8gYBJQxIWIR
--HG--
extra : rebase_source : eda4f601ba71380b41a1cc6182d21996d15ea4e6
This cleans up a few things, including simplifying the look of
backend.mk by keeping the relsrcdir in MERGE_RELATIVE_FILE similar to
the source path in the tree. Before, the locales/ floated around,
which is hard to understand but doesn't matter, since it's stripped by
MERGE_RELATIVE_FILE.
This also tests both relative and topsrcdir-absolute paths.
MozReview-Commit-ID: 1v3y9xGiNfL
--HG--
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/inner/locales/en-US/localized-input => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/locales/en-US/localized-input
extra : rebase_source : 302d7cb638974fc5ec71513f47ce98222c5e3bb6
`mach clobber python` currently purges Python files with known
extensions globally.
The python/ and third_party/python/ directories may also contain random
ignored/untracked files. Let's purge those as well.
Note: if someone has untracked but not ignored files, this will delete
them. This is possibly unwanted. But people shouldn't have untracked
but not ignored files sitting around in VCS. We don't run this command
by default, so I think it is safe to be aggressive in our purging of
these untracked files.
MozReview-Commit-ID: 8ql8QR6lP6j
--HG--
extra : rebase_source : 644eccee25913502ed4daa55e54aec9618ebe547
`mach clobber python` is supposed to remove autogenerated Python files.
Let's add .pyd files (compiled C extensions on Windows) to the list
for good measure.
MozReview-Commit-ID: EbHvBYB7hj3
--HG--
extra : rebase_source : 2736c3d1077c6f371df20b3854840612693117f8
There are a lot of choices and moving pieces in this commit. I elected
to include the mechanics and the target use case in the same commit so
that readers can compare and contrast the implementation and final
expression in one review window.
- Initially, I wanted to make the {AB_CD} substitutions in
LOCALIZED_FILES and not in LOCALIZED_GENERATED_FILES. However, I ran
into conceptual blockers doing this. Fundamentally, LOCALIZED_FILES
is FINAL_TARGET_FILES, and my use case should _not_ be putting files
anywhere near dist/bin. In addition, LOCALIZED_FILES
(FINAL_TARGET_FILES) is handled using manifests, which would need to
grow locale-aware functionality to handle this. That's not desirable.
In addition, if we use manifests, then we lose the powerful locality
of |mach build mobile/android{/base}| re-generating changed
locale-dependent resources. This is similar to how the build system
plumbs dist/idl manifest processing throughout the build: we're
repairing local workflows after moving work into a global process.
For these reasons, this doesn't support {AB_CD} in LOCALIZED_FILES.
- There is even another layer of complexity! There are two axes
involved with these files: AB_CD controls localization and the Make
target controls destination. For the record, it is:
regular builds - AB_CD unset
multi-locale builds - AB_CD set
single-locale repacks - AB_CD set
For the record, the existing logic (before any changes) is:
regular builds - Make target is `libs` in mobile/android/base/locales
multi-locale builds - Make target is `chrome-%` in mobile/android/base/locales
single-locale repacks - Make target is `libs` in mobile/android/base/locales
This commit adds targets for both destinations, and uses Make
chrome-%:: and libs:: magic to control what is invoked in the various
situations. Tricky!
- I added MERGE_RELATIVE_FILES in order to be able to follow-up this
patch with more patches that will get rid of
m/a/base/locales/{moz.build,Makefile.in} altogether, and fold this work
into m/a/base. As it stands, we're already reaching from
m/a/base/locales all the way out to
mobile/locales/.../region.properties, so the existing code doesn't
follow the layout expected between mozilla-central and
l10n-central/$(AB_CD). But that'll impedance will get worse as we
improve the build system dependencies, not better, so we should grow
support for localized resources that aren't exactly as expected.
- I chose to follow Python's syntax for string substitutions. I
would have preferred to mark files that should be localized with a
leading '%'... but I took that for filesystem absolute paths in
moz.build files already. I also considered @AB_CD@ to echo the
preprocessor, but didn't want to open the door to an expecation that
_all_ preprocessor DEFINEs will work in the way {AB_CD} does.
- The generate_*py script changes required a bit of a hack to "turn
off" locale dependent resources. This would have been nicer if we had
marked localized resources with '%'... but we didn't. See the
--fallback flag. The real reason this is needed is that we're doing
work which is more like the work of compare-locales (merging
locale-dependent resources) at build-time rather than repack time. I
don't know why that's the case -- probably when we (I) implemented it,
compare-locales and the whole l10n process was entirely opaque. It's
not worth changing it now, so we use this --fallback flag approach.
- I didn't get to tup support. This should gently fail without
breaking tup builds: any {AB_CD} substitutions just won't be
expanded. I haven't a clue how this should work in tup in the future
(or, more generally, how to make any sense of repacks without
declaring the full set of expected locales at configure time.)
- strings.xml can't be a LOCALIZED_PP_FILES, since we need to
customize the output location based on AB_rCD, and since we need a
little more flexibility than PP_FILES gives for our inputs.
MozReview-Commit-ID: MyfIkNSEzt
--HG--
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/en-US/localized-input => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/en-US/localized-input
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/foo-data => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/foo-data
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/generate-foo.py => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/generate-foo.py
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/en-US/localized-input => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/inner/locales/en-US/localized-input
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/moz.build => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/moz.build
rename : python/mozbuild/mozbuild/test/backend/data/localized-generated-files/non-localized-input => python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/non-localized-input
extra : rebase_source : 816b6f220758f2bb3bdd3ec81a2cb02269c6de5b
Without linking support in the tup backend, we don't produce a js
binary, so we can't copy it via OBJDIR_FILES yet.
MozReview-Commit-ID: AxqhHi84HIg
--HG--
extra : rebase_source : 1011bc6296be81646004a6bd6572193c34f367d8
Starting with Rust 1.24, the default codegen-units limit is 16,
with jobserver control to avoid overprovisioning. Remove our
previous fixed limit of 4 threads for debug builds.
For release, retain codegen-units=1 to make sure we get the
most complete optimization results.
Thanks to Simon Sapin for the suggestion.
MozReview-Commit-ID: FmYF4DcmBvt
--HG--
extra : rebase_source : 307ad8fad2874636adb3ce95a5cd47339e83f40c
The Proguard dependency is now managed by Gradle.
MozReview-Commit-ID: EOvKSE5z28P
--HG--
extra : rebase_source : 760b117f500cc639cc8c24e9c02933990f358dd7
The moz.build Java JAR definitions are, of course, broken, but they
will be removed soon enough.
MozReview-Commit-ID: KIxqLDwd9I7
--HG--
extra : rebase_source : 8312b3f125793f73d3e835d1c0a5c7cabd4ebc0c
I choose to clean a bunch of ANDROID_* moz.build cruft here, too,
since it's just passing dependencies between moz.build and
Makefile.in. The replacement for all of this is to just use
GENERATED_FILES in moz.build, but it'll still take some work to get to
that. (Why does this stuff exist? GENERATED_FILES didn't exist and
was resisted when I built this stuff.)
MozReview-Commit-ID: D3GJqJNL0Ih
--HG--
extra : rebase_source : 07351f9d3702cfc42c58bd317885d07882c45c3a
This is the easy stuff -- everything but mobile/android/base/Makefile.in.
MozReview-Commit-ID: 5x2z97AHUrR
--HG--
extra : rebase_source : 531fd41d367cad071b209b85ca5b5602fd7cbf7b
The last APK produced using the ANDROID_APK_* moz.build/Makefile.in
mechanism was Robocop, so we can get rid of these now.
MozReview-Commit-ID: 9b08ZvvOAoC
--HG--
extra : rebase_source : ac4fea057bf6e731b0f26a1b6902f17a7362076d
* Removed the revertNoRestartButton entry from the migration script as this no longer exists after this change
MozReview-Commit-ID: AGJ2OfYVPok
--HG--
extra : rebase_source : df7f383f51e4d72c493e4cc43696eff2ee0034fa
