glibc marks various allocation functions as `throw()`. This addition
hasn't been a problem until we tried to enable C++17, where clang
started complaining that we were redeclaring functions with mismatched
exception specifications. Peculiarly, glibc declares virtually
everything we redeclare as `throw()`, but clang only complains about the
mismatches for a particular subset of functions.
The approach taken in this patch is to add another potentially defined
macro to malloc_decls.h, `NOTHROW_MALLOC_DECL`. This macro works
exactly like `MALLOC_DECL`, except that clients can define
`NOTHROW_MALLOC_DECL` to add appropriate `throw()` specifiers when
declaring functions at global scope, and thereby avoid mismatched
exception specifications.
Differential Revision: https://phabricator.services.mozilla.com/D44045
--HG--
extra : moz-landing-system : lando
In the MinGW build, calls to malloc inside mozglue were not being
redirected as defined in the .def file. We create aliases for the
redirected functions to correctly redirect them inside mozglue.
An alternate solution for this exists. Rather than creating the
importlib during the linking step for mozglue, we could have used
dlltool to create it, and then provided it during linking. This
would allow mozglue to know that it should redirect calls to malloc
to je_malloc as specified in the .def file.
Differential Revision: https://phabricator.services.mozilla.com/D38407
--HG--
extra : moz-landing-system : lando
While here, properly export StackWalk.h when building with
--enable-project=memory.
Differential Revision: https://phabricator.services.mozilla.com/D40731
--HG--
extra : moz-landing-system : lando
None of the StringTable implementations were freeing their strdup'd
strings, either.
Differential Revision: https://phabricator.services.mozilla.com/D40757
--HG--
extra : moz-landing-system : lando
The most important part is the `GetPHCAddrInfo()` function in
`exception_handler.cc`, which extracts a crash address from an
`EXCEPTION_POINTERS` object.
The commit also changes the code so that it works if MozStackWalk() returns an
empty stack trace. This happens in practice on Windows on automation sometimes,
I'm not sure why.
The rest of the commit is just plumbing and the smoothing over of minor
platform differences.
Differential Revision: https://phabricator.services.mozilla.com/D39841
--HG--
extra : moz-landing-system : lando
- On Android, we were already doing it, but using fallible allocations.
- On *nix, it probably doesn't make a difference, but can't hurt. For
most things in Gecko, operator new/delete are inlined and thus
replaced by direct calls to the underlying allocator functions
(moz_xmalloc, malloc, etc.). This may have a benefit for some third
party libraries that would otherwise go through libstdc++'s to
eventually end up back into our allocator via the zone allocator
on macOS and via the exported symbols on others.
- On Windows, because of how some CRT static libraries are, a non-inlined
operator new (thanks to some disabled STL wrapping) would end up linked
against the system malloc, causing problems.
Overall, this can only be better. This also reduces the number of places
where we define those functions.
And on Android, this means operator new within mozglue becomes infallible,
which is more consistent with everything else.
Differential Revision: https://phabricator.services.mozilla.com/D36166
--HG--
extra : moz-landing-system : lando
Bug 1147248 added the workaround for GCC 4.9, but from an attempt with
GCC 6, it seems unnecessary anymore.
Differential Revision: https://phabricator.services.mozilla.com/D36165
--HG--
extra : moz-landing-system : lando
- On Android, we were already doing it, but using fallible allocations.
- On *nix, it probably doesn't make a difference, but can't hurt. For
most things in Gecko, operator new/delete are inlined and thus
replaced by direct calls to the underlying allocator functions
(moz_xmalloc, malloc, etc.). This may have a benefit for some third
party libraries that would otherwise go through libstdc++'s to
eventually end up back into our allocator via the zone allocator
on macOS and via the exported symbols on others.
- On Windows, because of how some CRT static libraries are, a non-inlined
operator new (thanks to some disabled STL wrapping) would end up linked
against the system malloc, causing problems.
Overall, this can only be better. This also reduces the number of places
where we define those functions.
And on Android, this means operator new within mozglue becomes infallible,
which is more consistent with everything else.
Differential Revision: https://phabricator.services.mozilla.com/D36166
--HG--
extra : moz-landing-system : lando
Bug 1147248 added the workaround for GCC 4.9, but from an attempt with
GCC 6, it seems unnecessary anymore.
Differential Revision: https://phabricator.services.mozilla.com/D36165
--HG--
extra : moz-landing-system : lando
This allows freelist randomization on a per-arena basis, by supplying parameters to
arena creation.
It uses an xorshift PRNG with a 128-bit state. It is not cryptographically secure. An
attacker who can observe outputs of the RNG, or read its state, is already in a position
to bypass the randomization applied. At the same time we make its state 128 bit to prevent
a trivial bypass if one or two outputs are observed.
The way a run selects masks to check has not been modified, so the randomization is limited
to at most 32 bits in the current mask being tested. It should be noted that while allocations
from the same run may now be non deterministic (up to the maximum entropy as previously
stated), an attacker who can perform multiple allocations will still be able to allocate
a targeted free region (for example while exploiting a use after free vulnerability in the
DOM). Non deterministic allocations will only impede an attacker who has less control over
how they allocate a targeted free region, and may provide some benefit during exploitation
of a heap based buffer overflow vulnerability where the attacker wishes to construct a
precise layout of regions pre overflow.
Differential Revision: https://phabricator.services.mozilla.com/D32219
--HG--
extra : moz-landing-system : lando
`jemalloc_replace_dynamic()` is badly broken. If you install a malloc table
other than the default at startup (e.g. DMD's or PHC's), when you call
`jemalloc_replace_dynamic()` it installs a new allocator that wraps the
*default* allocator, and then when you call `jemalloc_replace_dynamic(nullptr)`
it switches back to the *default* allocator.
This commits makes numerous improvements.
- It removes the "flip-flopping" between malloc tables, which didn't really
work and isn't necessary.
- `jemalloc_replace_dynamic()` now switches between the *original* malloc table
and the new one, rather than the *default* malloc table and the new one.
- It renames various things, to make the names shorter and clearer.
- It clearly documents the dangers and limitations of
`jemalloc_replace_dynamic()`.
- It removes and inlines `profiler::Init()`, because there was only one call
site.
- It rearranges `install_memory_counter()` so the control flow is simpler.
Differential Revision: https://phabricator.services.mozilla.com/D34266
--HG--
extra : moz-landing-system : lando
This makes it less mozjemalloc-specific, which is helpful for PHC. No non-test
code uses the extra detail anyway.
Differential Revision: https://phabricator.services.mozilla.com/D34441
--HG--
extra : moz-landing-system : lando
The current situation is suboptimal, where we have the same goop
repeated in multiple files, and where things kinda sorta work out fine
thanks to the linker for files that would have been forbidden, except
when the linker doesn't do its job, which apparently happen on
mingwclang builds.
This change only really covers C++ code using operator new/delete, and
not things that would be using malloc/free, because it's easier.
malloc/free is left for a followup.
Differential Revision: https://phabricator.services.mozilla.com/D32119
--HG--
extra : moz-landing-system : lando
MALLOC_STATIC_PAGESIZE is only set on some platforms. Specifically, it's
not set on ia64 and sparc. Which means the case MALLOC_STATIC_PAGESIZE
&& (sparc || ia64) never happens, and gPageSize is never 8 KiB.
Differential Revision: https://phabricator.services.mozilla.com/D31965
--HG--
extra : moz-landing-system : lando
In D25711, I added an arenaId member to `jemalloc_ptr_info_t` when `MOZ_DEBUG`
is defined. This modifies the GTest for `jemalloc_ptr_info()` to ensure that
the new member returns the correct value.
Differential Revision: https://phabricator.services.mozilla.com/D30087
--HG--
extra : moz-landing-system : lando
To ensure that any new JSString has its char buffer allocated in the new arena,
it is useful to be able to query a pointer and assert that it is in the
correct arena (at-least in Debug Build).
This adds the required functionality to mozjemalloc, and JSString can use it
for its new assertion in a later change.
Differential Revision: https://phabricator.services.mozilla.com/D25711
--HG--
extra : moz-landing-system : lando
Changes:
- added comments for tests being disabled
- disabled two additional tests in order to green the run
Differential Revision: https://phabricator.services.mozilla.com/D28085
--HG--
extra : moz-landing-system : lando
To ensure that any new JSString has its char buffer allocated in the new arena,
it is useful to be able to query a pointer and assert that it is in the
correct arena (at-least in Debug Build).
This adds the required functionality to mozjemalloc, and JSString can use it
for its new assertion in a later change.
Differential Revision: https://phabricator.services.mozilla.com/D25711
Changes:
- most tests are skipped using `moz.build` configuration file.
- `MultiWriterQueue` had to be skipped with `define` clauses in the test file due to build bustages when its `moz.build` file was used.
Differential Revision: https://phabricator.services.mozilla.com/D27944
--HG--
extra : moz-landing-system : lando
The jemalloc tests leave behind minidumps. Disable for now, for a green run.
Differential Revision: https://phabricator.services.mozilla.com/D27014
--HG--
extra : moz-landing-system : lando
This only copies the first `mLength` elements in `mPcs`.
Differential Revision: https://phabricator.services.mozilla.com/D21488
--HG--
extra : rebase_source : 91601501056fe7ea5b011adeca59033568fbde4a
This doesn't change behaviour because `sizeof(uintptr_t)` equals
`sizeof(uintptr_t*)`, but it's good to make things more obviously correct.
Differential Revision: https://phabricator.services.mozilla.com/D21487
--HG--
extra : rebase_source : 4df36d1d1e85a671c297e6e761b44b7da5b85343
Consequently, this removes:
- MOZ_LIBPRIO, which is now always enabled.
- non_msvc_compiler, which is now always true.
- The cl.py wrapper, since it's not used anymore.
- CL_INCLUDES_PREFIX, which was only used for the cl.py wrapper.
- NONASCII, which was only there to ensure CL_INCLUDES_PREFIX still
worked in non-ASCII cases.
This however keeps a large part of detecting and configuring for MSVC,
because we still do need it for at least headers, libraries, and midl.
Depends on D19614
Differential Revision: https://phabricator.services.mozilla.com/D19615
--HG--
extra : moz-landing-system : lando
***
Bug 1514594: Part 3a - Change ChromeUtils.import to return an exports object; not pollute global. r=mccr8
This changes the behavior of ChromeUtils.import() to return an exports object,
rather than a module global, in all cases except when `null` is passed as a
second argument, and changes the default behavior not to pollute the global
scope with the module's exports. Thus, the following code written for the old
model:
ChromeUtils.import("resource://gre/modules/Services.jsm");
is approximately the same as the following, in the new model:
var {Services} = ChromeUtils.import("resource://gre/modules/Services.jsm");
Since the two behaviors are mutually incompatible, this patch will land with a
scripted rewrite to update all existing callers to use the new model rather
than the old.
***
Bug 1514594: Part 3b - Mass rewrite all JS code to use the new ChromeUtils.import API. rs=Gijs
This was done using the followng script:
https://bitbucket.org/kmaglione/m-c-rewrites/src/tip/processors/cu-import-exports.jsm
***
Bug 1514594: Part 3c - Update ESLint plugin for ChromeUtils.import API changes. r=Standard8
Differential Revision: https://phabricator.services.mozilla.com/D16747
***
Bug 1514594: Part 3d - Remove/fix hundreds of duplicate imports from sync tests. r=Gijs
Differential Revision: https://phabricator.services.mozilla.com/D16748
***
Bug 1514594: Part 3e - Remove no-op ChromeUtils.import() calls. r=Gijs
Differential Revision: https://phabricator.services.mozilla.com/D16749
***
Bug 1514594: Part 3f.1 - Cleanup various test corner cases after mass rewrite. r=Gijs
***
Bug 1514594: Part 3f.2 - Cleanup various non-test corner cases after mass rewrite. r=Gijs
Differential Revision: https://phabricator.services.mozilla.com/D16750
--HG--
extra : rebase_source : 359574ee3064c90f33bf36c2ebe3159a24cc8895
extra : histedit_source : b93c8f42808b1599f9122d7842d2c0b3e656a594%2C64a3a4e3359dc889e2ab2b49461bab9e27fc10a7
Summary: Really sorry for the size of the patch. It's mostly automatic
s/nsIDocument/Document/ but I had to fix up in a bunch of places manually to
add the right namespacing and such.
Overall it's not a very interesting patch I think.
nsDocument.cpp turns into Document.cpp, nsIDocument.h into Document.h and
nsIDocumentInlines.h into DocumentInlines.h.
I also changed a bunch of nsCOMPtr usage to RefPtr, but not all of it.
While fixing up some of the bits I also removed some unneeded OwnerDoc() null
checks and such, but I didn't do anything riskier than that.
The diagnostic assert (so fortunately, it doesn't impact release builds)
as added in bug 1405159, but is costly because it uses the modulus of
the division with a variable integer, which is a slow operation.
However, in arena_run_reg_dalloc, we end up doing the same diagnostic
assert, in a different form: after performing the division in a faster
manner, we assert that the result, multiplied by the diviser, returns
the original number.
Differential Revision: https://phabricator.services.mozilla.com/D13501
--HG--
extra : moz-landing-system : lando
- modify line wrap up to 80 chars; (tw=80)
- modify size of tab to 2 chars everywhere; (sts=2, sw=2)
--HG--
extra : rebase_source : 7eedce0311b340c9a5a1265dc42d3121cc0f32a0
extra : amend_source : 9cb4ffdd5005f5c4c14172390dd00b04b2066cd7
So that dmd.py isn't broken by function names that, after stack fixing, contain
escape-worthy chars such as " and /.
--HG--
extra : rebase_source : 625cef56e4599bbb810273ef04af2b971e777b58
Previously the id for a new arena was just a counter that increased by one
every time. For hardening purposes, we want to make private arenas use a secure
random ID, so an attacker will have a more difficult time finding the memory
they are looking for.
Differential Revision: https://phabricator.services.mozilla.com/D10158
--HG--
extra : moz-landing-system : lando
Previously the id for a new arena was just a counter that increased by one
every time. For hardening purposes, we want to make the new counter a secure
random ID, so an attacker will have a more difficult time finding the memory
they are looking for.
Differential Revision: https://phabricator.services.mozilla.com/D10158
--HG--
extra : moz-landing-system : lando
JSONWriter currently calls new and delete indirectly through mozilla::MakeUnique to allocate a buffer. Becuase of this, the methods of this class cannot be invoked within Spidermonkey due to https://searchfox.org/mozilla-central/source/config/check_vanilla_allocations.py#6-14. Therefore, JSONWriter needs an AllocPolicy template parameter so that the allocation and deallocation routines can be changed to match the JS AllocPolicy when invoked within SpiderMonkey.
Differential Revision: https://phabricator.services.mozilla.com/D7279
--HG--
extra : moz-landing-system : lando
On some builds we only get 'SmokeDMD' references during testing rather than a full file path. This adds 'SmokeDMD' to the acceptable frame description list.
--HG--
extra : rebase_source : 475166fa6d1f241bf4d67cbdda1aab73000a5c3a
extra : intermediate-source : fcdc1d5eaf1a932992287074d0c91504dd7180d8
extra : source : 1192c15fc9347ccd6169dcb5bea81d7f2399e3fd
On some builds we only get 'SmokeDMD' references during testing rather than a full file path. This adds 'SmokeDMD' to the acceptable frame description list.
--HG--
extra : source : 1192c15fc9347ccd6169dcb5bea81d7f2399e3fd
extra : intermediate-source : d39c2dfcc7004a4736436c564de239ba68b20c03
extra : histedit_source : 5a75a20f5b88481c994597da81fdbc29186aba3d%2C3f60f2bfbe54ab3843d4b18d66296cd4aadf4a30
On some builds we only get 'SmokeDMD' references during testing rather than a full file path. This adds 'SmokeDMD' to the acceptable frame description list.
--HG--
extra : rebase_source : 2b1d2dbf8a3a538533554c6697f0e9ee51ec1bed
extra : source : 1192c15fc9347ccd6169dcb5bea81d7f2399e3fd
On some builds we only get 'SmokeDMD' references during testing rather than a full file path. This adds 'SmokeDMD' to the acceptable frame description list.
--HG--
extra : rebase_source : 147e62e22248971980d95fe211c86116a98cbdf4
Entry storage allocation now occurs on the first lookupForAdd()/put()/putNew().
This removes the need for init() and initialized(), and matches how
PLDHashTable/nsTHashtable work. It also removes the need for init() functions
in a lot of types that are built on top of mozilla::Hash{Map,Set}.
Pros:
- No need for init() calls and subsequent checks.
- No memory allocated for empty tables, which are not that uncommon.
Cons:
- An extra branch in lookup() and lookupForAdd(), but not in put()/putNew(),
because the existing checkOverloaded() can handle it.
Specifics:
- Construction now can take a length parameter.
- init() is removed. Explicit length-setting, when necessary, now occurs in the
constructors.
- initialized() is removed.
- capacity() now returns zero when the entry storage is absent.
- lookupForAdd() is no longer `const`, because it can instantiate the storage,
which requires modifications.
- lookupForAdd() can now return an invalid AddPtr in two cases:
- old: hashing failure (due to OOM in the hasher)
- new: OOM while instantiating entry storage
The existing failure handling paths for the old case work for the new case.
- clear(), finish(), and clearAndShrink() are replaced by clear(), compact(),
and reserve(). The old compactIfUnderloaded() is also removed.
- Capacity computation code is now in its own functions, bestCapacity() and
hashShift(). setTableSizeLog2() is removed.
- uint32_t is used throughout for capacities, instead of size_t, for
consistency with other similar values.
- changeTableSize() now takes a capacity instead of a deltaLog2, and it can now
handle !mTable.
Measurements:
- Total source code size is reduced by over 900 lines. Also, lots of existing
lines got shorter (i.e. two checks were reduced to one).
- Executable size barely changed, down by 2 KiB on Linux64. The extra branches
are compensated for by the lack of init() calls.
- Speed changed negligibly. The instruction count for Bench_Cpp_MozHash
increased from 2.84 billion to 2.89 billion but any execution time change was
well below noise.
We're currently using NDK r15c, which is rather old, and happens to come
with a buggy gold linker. Let's use a more recent NDK, with a fixed
linker.
Unfortunately, we're currently at NDK API level 9, which the newer NDK
doesn't provide for x86 anymore. But that corresponds to Gingerbread
(2.3), which we've long stopped supporting. On the SDK side, we already
dropped support of versions before Jelly Bean, so we can do the same on
the NDK side. That corresponds to API level 16. So let's just use that
as a baseline.
Another change in the newer NDK is that the target-name changed from
i386-linux-android to i686-linux-android, so adjust for that in the
android x86 mozconfigs.
