Split the GetCodeCoverageSummary API into a variant for a specific realm
vs checking all realms. This restores the original behaviour of the
getLcovInfo testing function to only return info on current realm. This
makes testing OOM behaviour much more predictable.
Differential Revision: https://phabricator.services.mozilla.com/D46167
--HG--
extra : moz-landing-system : lando
The code changes to use a profiler feature for the native allocations were deferred
until this commit.
Differential Revision: https://phabricator.services.mozilla.com/D42196
--HG--
extra : moz-landing-system : lando
The profiler backtraces inside a memory hook are re-entrant for the
profile lock. This patch adds additional guards to protect against this
in order to allow for a profiler backtrace during native allocations.
Differential Revision: https://phabricator.services.mozilla.com/D42195
--HG--
extra : moz-landing-system : lando
There is a bad interaction between the bloat log and the native allocation
tracking. There is a detailed message explaining the problem in the code.
Differential Revision: https://phabricator.services.mozilla.com/D42194
--HG--
extra : moz-landing-system : lando
This commit adds both the native allocation marker, and the Bernoulli
trial to attempt to sample the native allocations.
Differential Revision: https://phabricator.services.mozilla.com/D42193
--HG--
extra : moz-landing-system : lando
This commit adds a ThreadIntercept mechanism to block re-entry when
attempting to collect the stacks of native allocations.
A following commit will actually hook them up to collect allocations.
Differential Revision: https://phabricator.services.mozilla.com/D39882
--HG--
extra : moz-landing-system : lando
The ProfilerCounters had a static lifetime controlled by a UniquePtr. This
is an issue if the profiler mutex, which is also static, gets destroyed
first. The counters will then try to access the profiler mutex, and find
it already destroyed, leading to a crash. This commit changes the life
cycle to explicitly leak the counters for the lifteime of the process.
In addition, this patch adds explicit install and removal functions for
the memory hooks.
Differential Revision: https://phabricator.services.mozilla.com/D42192
--HG--
extra : moz-landing-system : lando
Profiling the profiler showed a lot of time spent allocating memory for the
frame-address strings, almost half the time of `StreamSamplesToJSON`!
Using an "Auto" string will prevent these allocations in most cases.
Also removed the `printf("0x%llx")`, instead just appending "0x" and an hex-
formatted number with `AppendInt()` (which handles 32- and 64-bit numbers
separately, so there is no more needs to do a double-cast to avoid sign
extensions -- There is still a double cast, but a no-op one to fix the type to
either `uint32_t` or `uint64_t`).
Using an Auto string for nearby frame labels as well.
Differential Revision: https://phabricator.services.mozilla.com/D45841
--HG--
extra : moz-landing-system : lando
Stack samples may contain up to hundreds or thousands of entries, which need to
be copied (during sampling and duplicating), and also mostly skipped when
creating the JSON output for other threads or other types of profile data.
This patch gathers all sample legacy entries (apart from the thread id and the
timestamp) into a single non-legacy entry, which is much faster to copy or
skip. The preceding timestamp now has a distinct Kind (`TimeBeforeCompactStack`)
to know whether to handle legacy entries of the new `CompactStack` entry kind.
Differential Revision: https://phabricator.services.mozilla.com/D45840
--HG--
extra : moz-landing-system : lando
profiler_can_accept_markers() is a fast and racy check that markers would
currently be stored, it should be used around potentially-expensive calls to
add markers.
And now markers are no longer stored when the profiler is paused. (Note that the
profiler is paused when a profile is being stored, this will help make this
operation faster.)
Differential Revision: https://phabricator.services.mozilla.com/D44434
--HG--
extra : moz-landing-system : lando
Since all profiler data is now stored inside ProfileBuffer, there is no real
need to continuously discard old data during sampling (this was particularly
useful to reclaim memory taken by old markers&payloads).
Instead, we can now just discard the old data once, just before starting to
stream it to JSON.
Differential Revision: https://phabricator.services.mozilla.com/D44433
--HG--
extra : moz-landing-system : lando
Now that what was in ProfilerMarker is stored directly in `BlocksRingBuffer`,
there is no need for this class anymore!
This also removes all the pointer management around it (when added to a TLS
list, moved during sampling, deleted when expired).
Differential Revision: https://phabricator.services.mozilla.com/D43431
--HG--
extra : moz-landing-system : lando
Since payloads are not kept alive long after they have been serialized, we can
just create them on the stack and pass a reference to their base (or pointer,
`nullptr` representing "no payloads") to `profiler_add_marker()`.
Differential Revision: https://phabricator.services.mozilla.com/D43430
--HG--
extra : moz-landing-system : lando
Markers and their payloads can now be serialized straight into the profiler's
`BlocksRingBuffer`, which is now thread-safe to allow such concurrent accesses
(even when gPSMutex is not locked).
This already saves us from having to allocate a `PayloadMarker` on the heap, and
from having to manage it in different lists.
The now-thread-safe `BlocksRingBuffer` in `CorePS` cannot be used inside the
critical section of `SamplerThread::Run`, because any mutex function could hang
because of the suspended thread (even though they functionally don't appear to
interact). So the sampler now uses a local `BlocksRingBuffer` without mutex.
As a bonus, the separate buffer helps reduce the number of concurrent locking
operations needed when capturing the stack.
Differential Revision: https://phabricator.services.mozilla.com/D43429
--HG--
extra : moz-landing-system : lando
Payloads will serialize themselves into a `BlocksRingBuffer` entry when first
captured.
Later they will be deserialized, to stream JSON for the output profile.
Differential Revision: https://phabricator.services.mozilla.com/D43428
--HG--
extra : moz-landing-system : lando
The common data members stored in the ProfilerMarkerPayload base class can be
gathered into a struct, which will make it easier to pass around, especially
when a derived object is constructed with these common properties.
Differential Revision: https://phabricator.services.mozilla.com/D43427
--HG--
extra : moz-landing-system : lando
Markers may contain backtraces, so we need to be able to de/serialize them.
This involves de/serializing the underyling `BlocksRingBuffer`.
Differential Revision: https://phabricator.services.mozilla.com/D43426
--HG--
extra : moz-landing-system : lando
The main `BlocksRingBuffer`s will be stored in `CorePS` (outside of
`ProfileBuffer`s), as we need to be able to safely access the underlying buffers
when profilers are not enabled.
Also `ProfilerBacktrace` will own the `BlocksRingBuffer` that its captured
`ProfileBuffer` uses.
Taking this opportunity to rename the different `mBuffer`s to more useful names.
Differential Revision: https://phabricator.services.mozilla.com/D43422
--HG--
extra : moz-landing-system : lando
Now that we are using a byte-oriented `BlocksRingBuffer` instead of an array of
9-byte `ProfileBufferEntry`'s, internally the profiler sets the buffer size in
bytes. However all external users (popup, tests, etc.) still assume that the
requested capacity is in entries!
To limit the amount of changes, we will keep assuming externally-visible
capacities are in entries, and convert them to bytes.
Even though entries used to be 9 bytes each, and `BlocksRingBuffer` adds 1 byte
for the entry size, some entries actually need less space (e.g., 32-bit numbers
now take 6 bytes instead of 9), so converting to less than 9 bytes per entry is
acceptable.
We are settling on 8 bytes per entry: It's close to 9, and it's a power of two;
since the effective number of entries was a power of two, and `BlocksRingBuffer`
also uses a power of two size in bytes, this convertion keeps sizes in powers of
two.
Differential Revision: https://phabricator.services.mozilla.com/D44953
--HG--
extra : moz-landing-system : lando
This just replaces `ProfileBuffer`'s self-managed circular buffer with a
`BlocksRingBuffer`.
That `BlocksRingBuffer` does not need its own mutex (yet), because all uses go
through gPSMutex-guarded code.
`ProfileBuffer` now also pre-allocates a small buffer for use in
`DuplicateLastSample()`, this avoids multiple mallocs at each sleeping thread
stack duplication.
Note: Internal "magic" sizes have been multiplied by 8 (and tweaked upwards, to
handle bigger stacks), because they originally were the number of 9-byte
entries, but now it's the buffer size in bytes. (And entries can now be smaller
than 9 bytes, so overall the capacity in entries should be similar or better.)
However, external calls still think they are giving a number of "entries", this
will be handled in the next patch.
Differential Revision: https://phabricator.services.mozilla.com/D43421
--HG--
extra : moz-landing-system : lando
Add some stats (off by default) around streaming JSON, as the following patches
may affect it.
Differential Revision: https://phabricator.services.mozilla.com/D44952
--HG--
extra : moz-landing-system : lando
In some situations we will *need* to use a `BlocksRingBuffer` that absolutely
does not use a mutex -- In particular in the critical section of
`SamplerThread::Run()`, when a thread is suspended, because any action on any
mutex (even a private one that no-one else interacts with) can result in a hang.
As a bonus, `BlocksRingBuffer`s that are known not to be used in multi-thread
situations (e.g., backtraces, extracted stacks for duplication, etc.) will waste
less resources trying to lock/unlock their mutex.
Differential Revision: https://phabricator.services.mozilla.com/D45305
--HG--
extra : moz-landing-system : lando
Profiling the profiler showed a lot of time spent allocating memory for the
frame-address strings, almost half the time of `StreamSamplesToJSON`!
Using an "Auto" string will prevent these allocations in most cases.
Also removed the `printf("0x%llx")`, instead just appending "0x" and an hex-
formatted number with `AppendInt()` (which handles 32- and 64-bit numbers
separately, so there is no more needs to do a double-cast to avoid sign
extensions -- There is still a double cast, but a no-op one to fix the type to
either `uint32_t` or `uint64_t`).
Using an Auto string for nearby frame labels as well.
Differential Revision: https://phabricator.services.mozilla.com/D45841
--HG--
extra : moz-landing-system : lando
Stack samples may contain up to hundreds or thousands of entries, which need to
be copied (during sampling and duplicating), and also mostly skipped when
creating the JSON output for other threads or other types of profile data.
This patch gathers all sample legacy entries (apart from the thread id and the
timestamp) into a single non-legacy entry, which is much faster to copy or
skip. The preceding timestamp now has a distinct Kind (`TimeBeforeCompactStack`)
to know whether to handle legacy entries of the new `CompactStack` entry kind.
Differential Revision: https://phabricator.services.mozilla.com/D45840
--HG--
extra : moz-landing-system : lando
profiler_can_accept_markers() is a fast and racy check that markers would
currently be stored, it should be used around potentially-expensive calls to
add markers.
And now markers are no longer stored when the profiler is paused. (Note that the
profiler is paused when a profile is being stored, this will help make this
operation faster.)
Differential Revision: https://phabricator.services.mozilla.com/D44434
--HG--
extra : moz-landing-system : lando
Since all profiler data is now stored inside ProfileBuffer, there is no real
need to continuously discard old data during sampling (this was particularly
useful to reclaim memory taken by old markers&payloads).
Instead, we can now just discard the old data once, just before starting to
stream it to JSON.
Differential Revision: https://phabricator.services.mozilla.com/D44433
--HG--
extra : moz-landing-system : lando
Now that what was in ProfilerMarker is stored directly in `BlocksRingBuffer`,
there is no need for this class anymore!
This also removes all the pointer management around it (when added to a TLS
list, moved during sampling, deleted when expired).
Differential Revision: https://phabricator.services.mozilla.com/D43431
--HG--
extra : moz-landing-system : lando
Since payloads are not kept alive long after they have been serialized, we can
just create them on the stack and pass a reference to their base (or pointer,
`nullptr` representing "no payloads") to `profiler_add_marker()`.
Differential Revision: https://phabricator.services.mozilla.com/D43430
--HG--
extra : moz-landing-system : lando
Markers and their payloads can now be serialized straight into the profiler's
`BlocksRingBuffer`, which is now thread-safe to allow such concurrent accesses
(even when gPSMutex is not locked).
This already saves us from having to allocate a `PayloadMarker` on the heap, and
from having to manage it in different lists.
The now-thread-safe `BlocksRingBuffer` in `CorePS` cannot be used inside the
critical section of `SamplerThread::Run`, because any mutex function could hang
because of the suspended thread (even though they functionally don't appear to
interact). So the sampler now uses a local `BlocksRingBuffer` without mutex.
As a bonus, the separate buffer helps reduce the number of concurrent locking
operations needed when capturing the stack.
Differential Revision: https://phabricator.services.mozilla.com/D43429
--HG--
extra : moz-landing-system : lando
Payloads will serialize themselves into a `BlocksRingBuffer` entry when first
captured.
Later they will be deserialized, to stream JSON for the output profile.
Differential Revision: https://phabricator.services.mozilla.com/D43428
--HG--
extra : moz-landing-system : lando
The common data members stored in the ProfilerMarkerPayload base class can be
gathered into a struct, which will make it easier to pass around, especially
when a derived object is constructed with these common properties.
Differential Revision: https://phabricator.services.mozilla.com/D43427
--HG--
extra : moz-landing-system : lando
Markers may contain backtraces, so we need to be able to de/serialize them.
This involves de/serializing the underyling `BlocksRingBuffer`.
Differential Revision: https://phabricator.services.mozilla.com/D43426
--HG--
extra : moz-landing-system : lando
The main `BlocksRingBuffer`s will be stored in `CorePS` (outside of
`ProfileBuffer`s), as we need to be able to safely access the underlying buffers
when profilers are not enabled.
Also `ProfilerBacktrace` will own the `BlocksRingBuffer` that its captured
`ProfileBuffer` uses.
Taking this opportunity to rename the different `mBuffer`s to more useful names.
Differential Revision: https://phabricator.services.mozilla.com/D43422
--HG--
extra : moz-landing-system : lando
Now that we are using a byte-oriented `BlocksRingBuffer` instead of an array of
9-byte `ProfileBufferEntry`'s, internally the profiler sets the buffer size in
bytes. However all external users (popup, tests, etc.) still assume that the
requested capacity is in entries!
To limit the amount of changes, we will keep assuming externally-visible
capacities are in entries, and convert them to bytes.
Even though entries used to be 9 bytes each, and `BlocksRingBuffer` adds 1 byte
for the entry size, some entries actually need less space (e.g., 32-bit numbers
now take 6 bytes instead of 9), so converting to less than 9 bytes per entry is
acceptable.
We are settling on 8 bytes per entry: It's close to 9, and it's a power of two;
since the effective number of entries was a power of two, and `BlocksRingBuffer`
also uses a power of two size in bytes, this convertion keeps sizes in powers of
two.
Differential Revision: https://phabricator.services.mozilla.com/D44953
--HG--
extra : moz-landing-system : lando
This just replaces `ProfileBuffer`'s self-managed circular buffer with a
`BlocksRingBuffer`.
That `BlocksRingBuffer` does not need its own mutex (yet), because all uses go
through gPSMutex-guarded code.
`ProfileBuffer` now also pre-allocates a small buffer for use in
`DuplicateLastSample()`, this avoids multiple mallocs at each sleeping thread
stack duplication.
Note: Internal "magic" sizes have been multiplied by 8 (and tweaked upwards, to
handle bigger stacks), because they originally were the number of 9-byte
entries, but now it's the buffer size in bytes. (And entries can now be smaller
than 9 bytes, so overall the capacity in entries should be similar or better.)
However, external calls still think they are giving a number of "entries", this
will be handled in the next patch.
Differential Revision: https://phabricator.services.mozilla.com/D43421
--HG--
extra : moz-landing-system : lando
Add some stats (off by default) around streaming JSON, as the following patches
may affect it.
Differential Revision: https://phabricator.services.mozilla.com/D44952
--HG--
extra : moz-landing-system : lando
