DMD currently uses a very hacky form of "sampling" by default to avoid
recording stack traces for all blocks. This makes DMD run faster than when it
records all stack traces.
This patch changes the sampling method used; in fact, it avoids "sampling" at
all. The existence of all heap blocks is now recorded exactly, but by default
we only record an allocation stack for each heap block if a Bernoulli trial
succeeds. This choice works well because getting the stack trace is ~100x
slower than recording the block's existence.
Overall, this approach is simpler and it also gives better output -- the choice
of which blocks to record allocation stacks for is mathematically sound, no
stack trace gets blamed for allocations it didn't do, and block counts and
sizes are now always exact.
Other specific things changed.
- All notion of sampling is removed from the various data structures.
- The --sample-below option is removed in favour of --stacks={partial,full}.
- The format of the JSON output file has changed.
- The names of various test files have changed to reflect concept changes.
--HG--
rename : memory/replace/dmd/test/full-empty-cumulative-expected.txt => memory/replace/dmd/test/complete-empty-cumulative-expected.txt
rename : memory/replace/dmd/test/full-empty-dark-matter-expected.txt => memory/replace/dmd/test/complete-empty-dark-matter-expected.txt
rename : memory/replace/dmd/test/full-empty-live-expected.txt => memory/replace/dmd/test/complete-empty-live-expected.txt
rename : memory/replace/dmd/test/full-unsampled1-dark-matter-expected.txt => memory/replace/dmd/test/complete-full1-dark-matter-expected.txt
rename : memory/replace/dmd/test/full-unsampled1-live-expected.txt => memory/replace/dmd/test/complete-full1-live-expected.txt
rename : memory/replace/dmd/test/full-unsampled2-cumulative-expected.txt => memory/replace/dmd/test/complete-full2-cumulative-expected.txt
rename : memory/replace/dmd/test/full-unsampled2-dark-matter-expected.txt => memory/replace/dmd/test/complete-full2-dark-matter-expected.txt
rename : memory/replace/dmd/test/full-sampled-live-expected.txt => memory/replace/dmd/test/complete-partial-live-expected.txt
extra : rebase_source : 47d287405dc5e9075f08addaba49e879c2c6e23f
Before, virtualenv.py may have attempted to use 3rd party
(untrusted) pip indices when installing wheels for pip,
setuptools, and wheel. These dependencies are vendored in
the tree for a reason. So don't let virtualenv contact the
outside world.
MozReview-Commit-ID: 6BCU0WegJO1
--HG--
extra : rebase_source : 6231235486e8cfa784e67e8b18004dd63d8aaf36
While we're addressing virtualenv foo, let's ensure we are running
the latest version. This also pulls in newer versions of pip (8.1.1),
setuptools (20.3), and wheel (0.29.0).
MozReview-Commit-ID: G5uSy66Kd6u
--HG--
extra : rebase_source : 804f230adcf77335c79a93537d9623ac3836d9bf
@depends functions are declared like the following:
@depends('--option', other_function, '--other-option', ...)
def ...
To simplify some of the processing related to those arguments it's
passed, keep a tuple of Option and DummyFunction objects corresponding
to those arguments.
For the same reasons as set_config is being moved to the global scope,
we're moving set_define to the global scope here. An additional change
is that set_define is now part of the sandbox itself instead of being
defined within the sandbox, which makes it share the implementation
details with set_config.
The way set_config is set currently makes it difficult to introspect
moz.configure files to know what configuration items are being set,
because they're hidden in the control flow of functions.
This makes some of the moz.configure more convoluted, but this is why
there are templates, and we can improve the recurring cases afterwards.
Currently, ConfigureSandbox._db stores two different kind of
information. This split those in two different instance variables
instead, making things clearer.
The way functions are being sandboxed in moz.configure land is that
their global namespace is being replaced with a limited and identifiable
dict. And we avoid re-wrapping a function that already received this
treatment.
The problem is that template functions have their global namespace
replaced, and any function that is defined within the template inherits
that global namespace. So when it comes time to wrap those functions
defined in templates with e.g. depends, we detect that they're already
wrapped although they are not, because we look if their global namespace
is of the recognizable type we use when replacing it.
So instead of looking at the global namespace type, keep track of all
functions that are wrapped.
Currently, if a @depends function doesn't have a return statement or
return None, a result is automatically set from all the set_config()
called from the function.
As we're going to move set_config to the global namespace, and as this
feature is only used once, and it's only used for something that was
written before ReadOnlyNamespace was exposed to the sandbox, we can
"safely" get rid of it.
This will be needed to teach artifact builds to extract files from
omni.ja files whose content is loaded into memory (from a tar
archive).
MozReview-Commit-ID: LH2HkKx5Zj3
--HG--
extra : rebase_source : 7f6b176f0ef9fc87889151d1d02da62de8a455d8
extra : source : 266928b5a7615fa054c70adf0f649cbb3f085e8d
This change adds a `Version` type to moz.configure which is a small
wrapper around `distutils.version.Version`. It's suitable for wrapping
version numbers in configure checks and doing equality or greater-than
less-than comparisons in a sensible way.
MozReview-Commit-ID: BOL6yvemulG
--HG--
extra : rebase_source : 3b463eac0499086f8acffda0d01418b6ab17f3d6
extra : amend_source : aebd6e40c408d9f868623b2f53fcdf7455e2fff5
In Python parens around an expression without a trailing comma is not a tuple,
so ('foo') == 'foo'. This is really easy to screw up with check_progs, which
coerces progs to a list and would give you ['f','o','o'] in this case. This
patch enforces that the progs argument is a tuple or list and errors if it
is not.
MozReview-Commit-ID: 7BJZuF9B8D5
--HG--
extra : rebase_source : 5db9a6b4bb9ef7195c2513407810093bff5e9174
extra : amend_source : f67ab46c2ac00a2a95cfc67e9763ac12b690ac14
If we add an ObjDirPath as an input to a GENERATED_FILES script, we
should run it in the misc tier to ensure the dependent files are created
beforehand. We need to use Path objects instead of raw filenames in the
GeneratedFile object so the recursive make backend can distinguish
between source and objdir files.
MozReview-Commit-ID: 9thHTi75zdI
While rare, this is something we support for e.g. --enable-eme, where it
can be either given with no values, "adobe", some future other EME GMP
adapter names, or a combination of them.
This will make it easier for binaries only archive generation to consult
the list of binaries that are relevant to packaging.
This does overlap somewhat with compile databases. Perhaps someday the
logic could converge.
While I was here, I cleaned up writing of the all-tests.json file to
avoid an intermediate string variable.
This patch adds to_dict methods on some frontend data types. There is
room to improve the serialization of these types. For example, we could
use __slots__ to drive the default formatter. We could also leverage a
custom JSONEncoder class that knows how to call to_dict() on instances.
In the spirit of perfect is the enemy of done, I think this should be
deferred to a follow-up bug.
MozReview-Commit-ID: XVnKB1MNqu
--HG--
extra : source : 4167dfdf10457360c9c94ee6e55b03ef1b92c16d
extra : histedit_source : 2f3f10bbb8ef9da8c57b0d85cbf4ea80242eff1a%2C1a5654253181d75eecd5dfaca2c87e353f8126cd
Now when we print instances in a debugger, we'll see something like:
<StaticLibrary: other-licenses/snappy/libother-licenses_snappy.a>
<StaticLibrary: toolkit/library/StaticXULComponentsEnd/libStaticXULComponentsEnd.a>
<SharedLibrary: toolkit/library/XUL>
<StaticLibrary: toolkit/library/libxul_s.a>
<HostProgram: config/nsinstall_real>
<Program: memory/replace/logalloc/replay/logalloc-replay>
<SimpleProgram: mfbt/tests/TestArrayUtils>
... instead of the the class name and memory address.
MozReview-Commit-ID: 8zdrM6KfP8U
--HG--
extra : source : 84849ad026c9ba1bbf71c93172b0a03440e51bec
This will make it easier for binaries only archive generation to consult
the list of binaries that are relevant to packaging.
This does overlap somewhat with compile databases. Perhaps someday the
logic could converge.
While I was here, I cleaned up writing of the all-tests.json file to
avoid an intermediate string variable.
This patch adds to_dict methods on some frontend data types. There is
room to improve the serialization of these types. For example, we could
use __slots__ to drive the default formatter. We could also leverage a
custom JSONEncoder class that knows how to call to_dict() on instances.
In the spirit of perfect is the enemy of done, I think this should be
deferred to a follow-up bug.
MozReview-Commit-ID: XVnKB1MNqu
--HG--
extra : rebase_source : 376c93467dde3450c7c21cff918cb34913a3c5fe
Now when we print instances in a debugger, we'll see something like:
<StaticLibrary: other-licenses/snappy/libother-licenses_snappy.a>
<StaticLibrary: toolkit/library/StaticXULComponentsEnd/libStaticXULComponentsEnd.a>
<SharedLibrary: toolkit/library/XUL>
<StaticLibrary: toolkit/library/libxul_s.a>
<HostProgram: config/nsinstall_real>
<Program: memory/replace/logalloc/replay/logalloc-replay>
<SimpleProgram: mfbt/tests/TestArrayUtils>
... instead of the the class name and memory address.
MozReview-Commit-ID: 8zdrM6KfP8U
--HG--
extra : rebase_source : 7f964ad44a0061674f77d5716a6769a4aedb9e6c
I'm not sure when this changed, but at least Visual Studio 2015
doesn't always emit a space between the line number and the ": warning"
text in cl.exe output.
Making the space optional in the regular expression enables one a
VS2015 build to capture 375 warnings instead of 17. We still fail to
capture some warnings (notably generic warnings about bad command
arguments and linker warnings). But that can be dealt with later.
MozReview-Commit-ID: q402CxTrQK
--HG--
extra : rebase_source : 6376a65b8d8145d68bad9b795b6abd6f4becefbd
Virtualenv will sometimes find a different executable from its sys.executable on OS X,
causing a check in the build system comparing filesizes between sys.executable and virtualenv
python to fail, resulting in clobbering and re-building the virtualenv every time the virtualenv
is activated, causing the build backend and more to be re-built.
Instead of checking file sizes directly, this commit causes us to record the size and version of the
Python executable that created the virtualenv. If the Python executable checked is not the virtualenv
Python, or we have a different version than was used to create the virtualenv, then the virtualenv is
considered to be out of date.
MozReview-Commit-ID: KmrVfQCtbS3
--HG--
extra : rebase_source : d2b87325e10da6dfcd74f8b0d2ef7c0efb71595b
Virtualenv will sometimes find a different executable from its sys.executable on OS X,
causing a check in the build system comparing filesizes between sys.executable and virtualenv
python to fail, resulting in clobbering and re-building the virtualenv every time the virtualenv
is activated, causing the build backend and more to be re-built.
Instead of checking file sizes directly, this commit causes us to record the size and path to the Python
executable that created the virtualenv. If the Python executable checked is not the virtualenv
Python, or the python that was used to create the interpreter, then the virtualenv is considered
to be out of date.
MozReview-Commit-ID: KmrVfQCtbS3
When writing e.g. Japanese text files on Windows, the text is encoded in
mbcs. So if a mozconfig is edited this way, and contains non ascii
characters, they won't be utf-8 as we're currently trying to decode
them, but mbcs.
This is currently preventing a command from having both args and subcommands at the same
time.
MozReview-Commit-ID: 66frAqamGjv
--HG--
extra : rebase_source : 3bbd1cb508e5aab7ba7f9936c0c486ed8f626fe9
