Further refine the order runs are returned by inserting fresh and other
runs in the middle of a size class' list leaving only decommitted and
madvised runs to be inserted at the end.
Differential Revision: https://phabricator.services.mozilla.com/D288828
We can prefer to allocate from dirty runs by inserting dirty runs at the
head of a size class' list and other runs at the end.
Differential Revision: https://phabricator.services.mozilla.com/D288827
This effectively backs out Bug 1980056 part 5 for a performance
regression. It also adds a comment about the performance regression.
This might be temporary depending on more testing, but for now we can
have some faster performance.
Differential Revision: https://phabricator.services.mozilla.com/D297236
The more regions (cells) in a run the more allocations can stay on the
"hot path". This can have a performance improvment on Linux and Windows
that corresponds to the larger page size on MacOS. On MacOS the larger
page size already implies more regions in a run.
Differential Revision: https://phabricator.services.mozilla.com/D297242
This effectively backs out Bug 1980056 part 5 for a performance
regression. It also adds a comment about the performance regression.
This might be temporary depending on more testing, but for now we can
have some faster performance.
Differential Revision: https://phabricator.services.mozilla.com/D297236
base_chunk_dealloc transitively re-enters BaseAlloc::alloc via chunk_record's
extent_node_t allocation. Since Mutex is non-recursive (Linux pthread adaptive,
macOS os_unfair_lock), this self-deadlocks on Linux and traps with SIGILL on
macOS. Capture the chunk under the lock and dealloc after dropping it; the
alloc path already takes the symmetric precaution via base_chunk_alloc.
Differential Revision: https://phabricator.services.mozilla.com/D299319
Replace the mRunsAvail RedBlackTree with a new class that uses an array
of linked lists. Inserting and removing runs are now constant time.
Searching for a run is linear with respect to the number of possible
sizes. The next patch will amortise that cost.
The previous RedBlackTree would bias towards returning runs with lower
memory addresses (when size is equal). That would make a slight
improvement to fragmentation. This patch may slightly increase
fragmentation, but the constant time operations on the tree makes up for
it.
Differential Revision: https://phabricator.services.mozilla.com/D288825
TLS and other features will need an array implementation. We can take
the one from PHC rename it and move it to a new header file.
Differential Revision: https://phabricator.services.mozilla.com/D279246
The chunk_allocator_t type is added to the public API of jemalloc and can
be set at the time of arena creation for private arenas. This type is
basically a virtual table that provides implementations for memory
map, unmap, commit and decommit. All arena chunk/page manipulations
now use the functions provided by its chunk allocator. By default this is a
global system allocator that preserves existing functionality.
The way this is currently implemented, an arena with a custom chunk allocator
will not have it's chunks recorded/recycled as before and would need to re-implement
that logic.
Differential Revision: https://phabricator.services.mozilla.com/D292240
chunk_record/chunk_recycle logic does not distinguish between
ARENA_CHUNK or HUGE_CHUNK in a meaningful way. It only checks
whether an incoming chunk is already ZEROED or not. So we can
treat all chunks passed to arena_chunk_dealloc as ARENA_CHUNK.
This change is mainly to allow the chunk allocator interface
introduced in the next change to not have to care about ChunkType
and avoid making ChunkType public.
Differential Revision: https://phabricator.services.mozilla.com/D292239
Introduce base_chunk_alloc and arena_chunk_alloc for base/arena chunk
allocations respectively. base_chunk_alloc skips the chunk recycling and
gChunkRTree insertion code paths as before.
Split up chunk deallocation into arena_chunk_dealloc and base_chunk_dealloc.
base_chunk_dealloc is used for both base chunks and arena chunks that have been
removed from gChunkRTree.
Differential Revision: https://phabricator.services.mozilla.com/D292238
The force_zero argument is unused. It is assumed that pages_purge
always zeroes pages so doesn't need to return a result anymore.
Besides the small cleanup this allows a future change that allows
arena chunk deallocs to be oblivious to the ARENA vs HUGE chunk distinction.
Differential Revision: https://phabricator.services.mozilla.com/D292236
The chunk_allocator_t type is added to the public API of jemalloc and can
be set at the time of arena creation for private arenas. This type is
basically a virtual table that provides implementations for memory
map, unmap, commit and decommit. All arena chunk/page manipulations
now use the functions provided by its chunk allocator. By default this is a
global system allocator that preserves existing functionality.
The way this is currently implemented, an arena with a custom chunk allocator
will not have it's chunks recorded/recycled as before and would need to re-implement
that logic.
Differential Revision: https://phabricator.services.mozilla.com/D292240
chunk_record/chunk_recycle logic does not distinguish between
ARENA_CHUNK or HUGE_CHUNK in a meaningful way. It only checks
whether an incoming chunk is already ZEROED or not. So we can
treat all chunks passed to arena_chunk_dealloc as ARENA_CHUNK.
This change is mainly to allow the chunk allocator interface
introduced in the next change to not have to care about ChunkType
and avoid making ChunkType public.
Differential Revision: https://phabricator.services.mozilla.com/D292239
Introduce base_chunk_alloc and arena_chunk_alloc for base/arena chunk
allocations respectively. base_chunk_alloc skips the chunk recycling and
gChunkRTree insertion code paths as before.
Split up chunk deallocation into arena_chunk_dealloc and base_chunk_dealloc.
base_chunk_dealloc is used for both base chunks and arena chunks that have been
removed from gChunkRTree.
Differential Revision: https://phabricator.services.mozilla.com/D292238
The force_zero argument is unused. It is assumed that pages_purge
always zeroes pages so doesn't need to return a result anymore.
Besides the small cleanup this allows a future change that allows
arena chunk deallocs to be oblivious to the ARENA vs HUGE chunk distinction.
Differential Revision: https://phabricator.services.mozilla.com/D292236
When freeing cells, if the cell has no neighbours after merging then it
is an entire chunk. Release the whole chunk to the OS.
Differential Revision: https://phabricator.services.mozilla.com/D287069
When cells are freed they are merged with their free neighbours, unless
those neighbours are decommitted which can lead to fragmentation.
Merging these cells during free() would mean committing one or
decommitting the other. Instead if an allocation can't be satisfied by
the normal or oversize free lists, process the decommited free list (a
tree) to commit then merge small decommitted cells with their neighbours
until a cell large enough to satisfy the allocation is made available.
This solves a test failure on Windows.
Differential Revision: https://phabricator.services.mozilla.com/D287068
When decommitting memory split cells of the beginning and end of the
decommitted region so that the maximum amount of committed memory is
available.
Differential Revision: https://phabricator.services.mozilla.com/D270990
Instead when BaseAlloc needs more memory from the system it converts a
whole chunk into a cell then splits it. This will make it easier to
release whole chunks later without needing to manage "wilderness".
This has the side-effect of always keeping all memory committed. A
later patch will add the ability to decommit memory.
Differential Revision: https://phabricator.services.mozilla.com/D270988
+ The internal allocation functions now return BaseAllocCell pointers.
* The alloc size checks and allocation state are now done in one
place, when alloc_cell() returns control to alloc().
+ The old wilderness_alloc function is removed and
wilderness_alloc_inplace is renamed to wilderness_alloc
+ Add log messages to alloc and free functions.
Differential Revision: https://phabricator.services.mozilla.com/D270987
Introduce a "quantum" cell size so that free lists only ever contain
objects of the same size which improves cell reuse.
Differential Revision: https://phabricator.services.mozilla.com/D268991
The next patch will need to access some of these structures from
BaseAlloc.h, rather than adding them there and making the interface less
obvious, create a new header.
Differential Revision: https://phabricator.services.mozilla.com/D268989
Paul Bone