Instead of waiting a set time guessed from how long the parent process took to do its work, after a short time the parent process requests progress updates from all still-pending child processes, and restarts the timer if any progress was made.
If processes become unresponsive, they will be the last ones pending, and after one timer cycle without any progress anywhere, the parent process won't wait for children anymore, and will output all profiles successfully gathered so far.
Added `MOZ_LOG=prof` logging in nsProfiler.cpp, to monitor profile-gathering. (And removed a spurious 'd' character in the `LOG` macro.)
Differential Revision: https://phabricator.services.mozilla.com/D135488
This code will be used again in the following patch.
Instead of destroying and re-creating a new timer, we can re-initialize the existing one.
Also add the timer name "nsProfilerGatheringTimer" when first creating it.
Differential Revision: https://phabricator.services.mozilla.com/D135487
This helper function in ProfilerParent sends a progress request to a child process. If successfully sent, the response will resolve the returned promise.
Differential Revision: https://phabricator.services.mozilla.com/D135486
A new IPC function allows the parent process to request a progress update from any child process.
If a profile generation is in progress, the shared `ProportionValue` can be atomically read and sent back in response.
Differential Revision: https://phabricator.services.mozilla.com/D135485
In order to keep the child process responsive to profile IPCs, the heavy work of generating the profile JSON is now done in a separate thread.
A `ProgressLogger` is used to keep track of the progress of this work, and the progress value is stored in a shared atomic `ProportionValue`.
When the JSON profile is ready, the final shmem allocation (used to send the profile to the parent process) is done on the original "ProfilerChild" IPC thread.
Differential Revision: https://phabricator.services.mozilla.com/D135484
The main goal is to separate the profile generation (in a JSONWriter) from the final allocation needed to output the profile in one block.
This will be needed in the next patch, where the profile generation will be done in a new worker thread, but the shmem allocation *must* be done on the original "ProfilerChild" thread that handles IPC responses.
Differential Revision: https://phabricator.services.mozilla.com/D135483
Instead of just waiting for a certain number of profiles, the parent process now waits for profiles from a predetermined list of child process ids.
When receiving a profile, or when something goes wrong with a child process, the corresponding listed id can be removed, until the list is empty.
In a later patch, this list will be used to request progress updates from slow processes.
Differential Revision: https://phabricator.services.mozilla.com/D135482
This will be useful to tie profiles to the child process id that generated them. (At the moment, the parent waits for a number of profiles, but doesn't check where received profiles actually come from.)
Differential Revision: https://phabricator.services.mozilla.com/D135481
A small optimization while working on nearby code, so avoid multiple allocations when we already know how much memory we really need.
Differential Revision: https://phabricator.services.mozilla.com/D135480
Add `ProgressLogger` parameter to most JSON-generating functions.
Each function can update the given `ProgressLogger` between 0% and 100%, and create sub-loggers when calling functions.
The main goal of this instrumentation is to notice when any progress is made by child processes (when the parent process is gathering profiles), so it needs to go deep enough so that it is not stuck on a progress value for "too long" -- During development, that meant progress was always happening when observed every 10ms; In later patches, the overall timeout for no-progress-made will be at least 1 second.
Differential Revision: https://phabricator.services.mozilla.com/D135479
Rather than defining a distinct `Parameter` class for Gecko, this now uses the
`extend_parameters_schema` utility function (which mobile repos are already
using).
As a consequence, shared parameters are now defined in standalone taskgraph.
And only Gecko-specific parameters are listed in
`gecko_taskgraph/parameters.py`
The only exception is `project` which gets redefined so we can override the
standalone taskgraph default (since it derives `project` from the repo name,
which doesn't work for Gecko).
Differential Revision: https://phabricator.services.mozilla.com/D134515
As mentioned in bug 1747354, the location of the dist directory is
relied to be $topobjdir/dist, so just use that consistently rather
than getting it from a separate variable for rust build scripts.
Differential Revision: https://phabricator.services.mozilla.com/D136556
As mentioned in bug 1747354, the location of the dist directory is
relied to be $topobjdir/dist, so just use that consistently rather
than getting it from a separate variable for rust build scripts.
Differential Revision: https://phabricator.services.mozilla.com/D136556
In rare cases, a label may be left dangling, or would be removed too late (after the thread unregisters itself).
The most common cause is from Windows DLL load labels, which contain important information that we want to keep, so it's worth letting the registration data leak in this case.
Later bugs will soon fix the issue in a better way, and remove this leak.
Differential Revision: https://phabricator.services.mozilla.com/D136700
A bunch of modern packages (`pytest`, `twisted`, `automat`) all need
`attrs==19.2.0` (or newer).
We _could_ bump `attrs` all the way to the modern `21.4.0` version, but
I'd like to defer that upgrade risk, since there's a
lot of backwards-incompatible changes and deprecations. So, lightly bump
it to `19.2.0`.
As part of bumping it, `pytest` is no longer compatible.
The earliest candidate that seems to be compatible is `pytest` 4.6.6,
which boasts in its release notes that it's resolved some deprecation
warnings against `attrs>=19.2.0`.
Once `pytest` was bumped, it needed a newer version of `pluggy`, which
itself has dependencies.
Since we're using hashes in `tox_requirements.txt`, all dependencies
needed to be hashed as well.
Differential Revision: https://phabricator.services.mozilla.com/D135178
Note: The atomic variable is named `gSkipSampling`, not mentioning forks because it could later be used in other situations, so it's best to describe it by the effect it has.
Differential Revision: https://phabricator.services.mozilla.com/D136205
Ideally 'manifest' and 'manifest_relpath' should be normalized to forward
slashes upstream in the TestResolver class. But normalizing them there could
potentially break other uses in-tree, and I don't have bandwidth to do a proper
audit to be confident I'm not breaking something elsewhere.
Differential Revision: https://phabricator.services.mozilla.com/D136183
This helps when dealing with threads that are not registered, e.g.: the Java thread, and the upcoming whole-process thread. And it removes some object copies.
Differential Revision: https://phabricator.services.mozilla.com/D133172
`PrintUsageThenExit(code)` was supposed to exit when `code` was not zero, but:
- The name didn't reflect that, so it was confusing that `PrintUsageThenExit(0)` would *not* exit.
- The implementation in the Base Profiler exited anyway! This caused issues with some legacy code that still used the now-removed "threads" feature.
This patch renames the function to just `PrintUsage()` and never exits, leaving the caller to invoke `exit(code)` as needed -- with the added benefit that it's possible to exit with a zero code, useful in cases where an exit is not actually an error.
Differential Revision: https://phabricator.services.mozilla.com/D135666
This generates a list of all known targets in an existing build of the
docs. That makes it easier/possible to figure out what references
exist and can be used.
Differential Revision: https://phabricator.services.mozilla.com/D135596
The idea is to capture the warnings in a temporary file, and then
apply a set of regex to find any that should be treated as fatal.
This allows us to fix warnings one type at a time, and prevents us
regressing the warnings that are already fixed.
The "reference target not count" warning is added to the initial
forbidden list, so we can ensure we don't end up with internal links
pointing to nowhere.
Differential Revision: https://phabricator.services.mozilla.com/D135389
