Bug 1300974 - Work around race condition leading to deadlock on fork when enabling LogAlloc. r=njn

memory/replace/logalloc/LogAlloc.cpp

@@ -86,7 +86,28 @@ replace_init(const malloc_table_t* aTable)
    * other thread is holding the lock before forking, by acquiring it
    * ourselves, and releasing it after forking, both in the parent and child
    * processes.
-   * Windows doesn't have this problem since there is no fork(). */
+   * Windows doesn't have this problem since there is no fork().
+   * The real allocator, however, might be doing the same thing (jemalloc
+   * does). But pthread_atfork `prepare` handlers (first argument) are
+   * processed in reverse order they were established. But replace_init
+   * runs before the real allocator has had any chance to initialize and
+   * call pthread_atfork itself. This leads to its prefork running before
+   * ours. This leads to a race condition that can lead to a deadlock like
+   * the following:
+   *   - thread A forks.
+   *   - libc calls real allocator's prefork, so thread A holds the real
+   *     allocator lock.
+   *   - thread B calls malloc, which calls our replace_malloc.
+   *   - consequently, thread B holds our lock.
+   *   - thread B then proceeds to call the real allocator's malloc, and
+   *     waits for the real allocator's lock, which thread A holds.
+   *   - libc calls our prefork, so thread A waits for our lock, which
+   *     thread B holds.
+   * To avoid this race condition, the real allocator's prefork must be
+   * called after ours, which means it needs to be registered before ours.
+   * So trick the real allocator into initializing itself without more side
+   * effects by calling malloc with a size it can't possibly allocate. */
+  sFuncs->malloc(-1);
   pthread_atfork(prefork, postfork, postfork);
 #endif