Generating the list of idl deps to generate an xpt from its dependency list
makes us give all _previous_ dependencies, inherited from the .deps makefiles.
This leads to removed files being listed on xpidl-process.py command line, and
the command subsequently failing.
Instead, use generated lists of idl dependencies. At the same time, lighten the
generated Makefile further by not emitting xpt dependencies on their containing
directory, and instead generating it from the $xpt_files list.
This brings down the Makefile size from 100k to 38k.
Similar to the changes made for IPDL files, this commit moves all of the
non-makefile related logic for WebIDL files out of the recursive make
backend and into the common build backend. Derivative backends that
would like to do interesting things with WebIDL files now need to
implement _handle_webidl_build, which takes more parameters, but should
ideally require less duplication of logic.
After a bunch of tiny changes, we're finally ready to make real
progress. We can now move the grouping of the generated IDPL C++ files
and the actual writing of the unified files for them into the common
build backend. Derivative backends now only have to concern themselves
with adding the particular logic that compiling those files requires.
We'll need to write out unified files for multiple backends, not just
the recursive make one. Put that logic someplace where all build
backends can access it.
_add_unified_build_rules shouldn't be in the business of determining how
to group files into their unified files. That logic belongs in the
caller of _add_unified_build_rules. Once that's done, the logic for
determining how to group files can migrate out of the recursive make
bakend.
Nothing about writing unified files is specific to the recursive make
backend, and if we want to write the unified files for IPDL and WebIDL
files, we'll need this functionality available in the common build
backend.
RecursiveMakeBackend._group_unified_files doesn't contain any
functionality specific to the recursive make backend. We would also
like to move the unification of generated IPDL and WebIDL source files
into the common build backend. Moving _group_unified_files into the
common build backend would be the logical place for it, but the frontend
should also be able to handle unifying files so that backends don't have
to duplicate logic for UNIFIED_FILES. Therefore, we choose to move it
to mozbuild.util as its final resting place.
Pushing on a CLOSED TREE since Android build only.
--HG--
extra : rebase_source : cc99efa734d1f4738d3f026c930a4d1955723783
extra : amend_source : 52c07807a77f263d2eb2593826dc0285928d9be4
This handles:
list.0=A
list.1=B
list.sublist.0=C
so that
list=>[A, B]
list.sublist=>[C]
and
dict=default
dict.key1=A
dict.key2=B
so that
dict=>{key1:A, key2:B}
dict=>default
--HG--
extra : rebase_source : 2c8f5941d2fca9c56b3858d3e98078a43fa69090
Writing the unified files is another thing that will have to be moved
out of recursivemake.py eventually. And it doesn't belong inline amidst
makefile rules and variables. Move its logic to a separate function as
well.
_add_unified_build_rules does quite a lot of work besides adding
makefile rules and variables. The divying up of source files into
unified files is one part of that, so move it out into its own function.
When we eventually move that computation out of recursivemake.py, this
refactoring will make it easier to verify that's what we've done.
Python API documentation requires the ability to import modules. So, we
set up a virtualenv in our Sphinx environment so module loading works.
This solution isn't perfect: a number of modules fail to import when run
under sphinx-build.
--HG--
extra : rebase_source : fce732e0b8aefe0e9a2ee594b8a08ac02e27579a
extra : histedit_source : bef27c947b95c3182fbc6cd656ae8c96acaaa6f4
Previously, code for staging the Sphinx documentation from moz.build
metadata lived in a mach command and in the moztreedocs module. This
patch moves the invocation to the Sphinx extension.
When the code is part of the Sphinx extension, it will run when executed
with sphinx-build. This is a prerequisite to getting RTD working, since
sphinx-build is the only supported entrypoint for generating
documentation there.
With this patch, we can now invoke sphinx-build to build the
documentation. The `mach build-docs` command is no longer needed.
--HG--
extra : rebase_source : 86e76c7d598ffa23dae858254eecedbdd12706a4
extra : histedit_source : 1826aa5ddfafdff62847cc293d1f0950b236caac
The recursivemake backend knows how to do several things with the IPDL
sources:
1) Determine the C++ sources that will be generated from given IPDL
sources.
2) Write out all the makefile rules and variables for said sources.
The first part isn't unique to the recursivemake backend; other backends
would eventually like to know what C++ sources come from IPDL source
files for easier cross-referencing purposes, etc. Let's take a first
cut at moving things into CommonBackend. (This may not be the best
interface, since it relies on consume_finished being invoked, and not
all backends call CommonBackend.consume_finished. Still, it's a start.)
Various bits of the test harnesses key off of mozinfo.info.get('asan');
we will need a similar switch for finding out whether this build
supports tsan.
Now that the mozbuild backend knows about FINAL_TARGET, we are able to
install generated xpt files into their final location. This saves us
from copying xpt files into their final location on every build.
Original patch by gps, rebased and comments addressed by Ms2ger
--HG--
extra : transplant_source : %E2%DC%0F%E0%AD%C2%25%A1%B8%A9%FE%B0%8C%60%FF%CB%02G%25%E5
mopack.BaseFile.copy() performs a generic read/write file copy. Windows
has an explicit CopyFile() call that tests have shown to be
significantly faster. Let's use that instead via the magic of ctypes.
Having SOURCES and its close relatives go through VariablePassthru
objects clutters the handling of VariablePassthru in build backends and
makes it less obvious how to handle things that actually get compiled.
Therefore, this patch introduces four new moz.build objects
corresponding to the major variants of SOURCES. It looks like a large
patch, but there's an ample amount of new tests included, which accounts
for about half of the changes.
Now that defining $DMD is no longer necessary to run DMD, this patch does the
following.
- Removes all the places where we set DMD=1 (test harnesses, etc.)
- Still handles DMD=1, for backwards compatibility.
- Prints "$DMD is undefined" at DMD start-up if appropriate.
- Writes a |null| value for |dmdEnvVar| in the JSON if $DMD is undefined. Bumps
the DMD output version number accordingly.
- Changes a bunch of the test files accordingly, including changing the mode of
script-ignore-alloc-fns.json in order to test a case where $DMD is undefined.
--HG--
extra : rebase_source : eb1ef5722410734ce6d7658465ff6f442ee4ed49
Various os.path attributes are being used in tight loops. Having local
variables prevents extra dictionary lookups.
This appears to shave 10-20ms off of the tests install manifest
processing time.
--HG--
extra : rebase_source : de941f2978cf0b1fd7c4f7401c848b61d406a2c8
extra : amend_source : e33c896856fa559197496b8227e10ab8149d146e
FileCopier.copy() was performing a lot of os.path.normpath() operations.
Profiling revealed that os.path.normpath() was the function with the
most wall time CPU usage when processing the tests manifests. Upon
subsequent examination of the code in question, all the paths being used
were already normalized. So, os.path.normpath() wasn't accomplishing
anything.
This patch results in ~300ms reduction in wall time to process the tests
install manifest on a fully populated page cache. Execution time drops
from ~2.8s to ~2.5s.
Profiling reveals that after this patch os.stat() is the #1 wall time
consumer. However, os.path.{join,dirname,normpath} still account for
~1.5x the wall time of os.stat(). There is still room to optimize
this function.
--HG--
extra : rebase_source : b6f0862baa5168c609499fd95eb3517854bc8cce
extra : amend_source : 7e04c1eb74132bbbe86e721f0f209b19309a7a51
This patch moves profiling mode selection from post-processing (in dmd.py) to
DMD start-up. This will make it easier to add new kinds of profiling, such as
cumulative heap profiling.
Specifically:
- There's a new --mode option. |LiveWithReports| is the default, as it is
currently.
- dmd.py's --ignore-reports option is gone.
- There's a new |mode| field in the JSON output.
- Reports-related operations are now no-ops if DMD isn't in LiveWithReports
mode.
- Diffs are only allowed for output files that have the same mode.
- A new function ResetEverything() replaces the SetSampleBelowSize() and
ClearBlocks(), which were used by the test to change DMD options.
- The tests in SmokeDMD.cpp are split up so they can be run multiple times, in
different modes. The exact combinations of tests and modes has been changed a
bit.
--HG--
rename : memory/replace/dmd/test/full-reports-empty-expected.txt => memory/replace/dmd/test/full-empty-dark-matter-expected.txt
rename : memory/replace/dmd/test/full-heap-empty-expected.txt => memory/replace/dmd/test/full-empty-live-expected.txt
rename : memory/replace/dmd/test/full-heap-sampled-expected.txt => memory/replace/dmd/test/full-sampled-live-expected.txt
rename : memory/replace/dmd/test/full-reports-unsampled1-expected.txt => memory/replace/dmd/test/full-unsampled1-dark-matter-expected.txt
rename : memory/replace/dmd/test/full-heap-unsampled1-expected.txt => memory/replace/dmd/test/full-unsampled1-live-expected.txt
rename : memory/replace/dmd/test/full-reports-unsampled2-expected.txt => memory/replace/dmd/test/full-unsampled2-dark-matter-expected.txt
rename : memory/replace/dmd/test/script-diff-basic-expected.txt => memory/replace/dmd/test/script-diff-live-expected.txt
rename : memory/replace/dmd/test/script-diff1.json => memory/replace/dmd/test/script-diff-live1.json
rename : memory/replace/dmd/test/script-diff2.json => memory/replace/dmd/test/script-diff-live2.json
extra : rebase_source : bf32cc4e0d82aa1a20ceb55e8ea259850b49cc06
This is a straight copy from
a878bf0ba0
paired with a tiny change to use the new quote_chars option.
--HG--
extra : rebase_source : 75d604ffafc7062c663bca4242af35546d2c1e3a
buildlist invocations are slow and can occur in parallel since the
underlying program obtains a lock on the modified file.
Moving the XPT-related buildlist invocation from the serial libs tier to
the parallel misc tier decreased my no-op build time on OS X from 43.5s
to 37.0s.
--HG--
extra : rebase_source : 7d274024c401b1ecfbc771424a69eb487808fcbf
