Bug 1902213 - Add Pernosco documentation r=gcp

Differential Revision: https://phabricator.services.mozilla.com/D214626

docs/contributing/debugging/debugging_firefox_with_pernosco.rst

@@ -0,0 +1,101 @@
+Debugging Firefox with Pernosco
+===============================
+
+This page is intended to help Firefox/Gecko developers get started using `Pernosco <https://pernos.co/>`_
+to debug Firefox.
+
+Mozilla employees have a paid subscription to Pernosco, and are encouraged to use it. 
+Contributors can use Pernosco as well. `A free account <https://pernos.co/pricing/>`_ 
+comes with 5 uploads to see if it is suitable for your workflow. 
+
+Prerequisites
+-------------
+
+- You need the ability to produce an :code:`rr` trace. See :doc:`debugging_firefox_with_rr`
+- Employees, you can also generate traces for Pernosco for some CI job failures via
+  Treeherder.
+
+Workflow
+--------
+
+At a high level, working with Pernosco consists of recording a trace then uploading 
+the trace for processing. 
+
+Install your Pernosco configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are using your own Pernoso account, follow the instructions 
+`here <https://pernos.co/account>`_ to install your credentials. 
+
+Mozilla employees, see the `Pernosco install and configuration doc 
+<https://docs.google.com/document/d/1cb-mkNyJYMudpS0a-R6zxy0kOMIRUSKrxxL-E9zFGKE/edit>`_
+for credentials. 
+
+In general, you will want: 
+
+ * Your email in :code:`~/.config/pernosco/user`
+ * Your group in :code:`~/.config/pernosco/group`
+ * Your key in :code:`~/.config/pernosco/user_secret_key`
+
+Gather a Local Trace
+~~~~~~~~~~~~~~~~~~~~
+
+In general, to gather a trace follow the instruction for recording :code:`rr` traces 
+in :doc:`debugging_firefox_with_rr`. However, to gather a local rr trace that’s 
+compatible with Pernosco you need to disable some incompatible CPU features (currently
+only AVX-512)
+
+On an Intel machine, you can disable this at the command line. To simplify things a 
+shell script :code:`pernosco-record` can be created like so, 
+
+.. code:: bash
+
+    #!/bin/bash
+    
+    rr record --disable-avx-512 "$@"
+
+replacing plain :code:`rr` in command lines (:code:`mach --debugger pernosco-record`)
+
+On AMD Zen Machines, you can disable AVX-512 and 5 level page tables at boot.
+
+On a Debian derived distro this involves editing :code:`/etc/default/grub` and 
+adding :code:`clearcpuid=304,no5lvl` to :code:`GRUB_CMDLINE_LINUX_DEFAULT`, then 
+running :code:`sudo update-grub` after. 
+
+Upload the trace
+~~~~~~~~~~~~~~~~
+ 
+Find the trace you’re interested in :code:`~/.local/share/rr/<trace>`, and call :code:`pernosco-submit`
+
+.. code:: bash 
+
+   pernosco-submit upload ~/.local/share/rr/<trace> <PATH TO CENTRAL> --title "Name Of Trace"
+
+where :code:`<PATH TO CENTRAL>` is the path to the source of your build, checked out
+to that revision.
+
+The last trace recorded is always symlinked, so you can use :code:`~/.local/share/rr/latest-trace` too. 
+
+Wait for email
+~~~~~~~~~~~~~~
+
+After the trace is processed, you'll get an email with a link to your trace. Good 
+hunting!
+
+Treeherder Workflow
+-------------------
+
+You can also try to get traces generated via the Treeherder interface. On a failing 
+test that is eligible, in "Artifacts and Debugging Tools", you'll see a link to 
+
+.. image:: ../img/record-this-failure.png 
+    :width: 35 %
+
+This link brings you to the self-service reproducer: 
+
+.. image:: ../img/self-service.png 
+    :width: 60 %
+
+Select a test and a count, and whether or not chaos-mode should be used, then click 
+reproduce. At this point there is a **long** wait; the page will notify you when 
+reproduction has started with an :code:`alert` dialog. 

docs/contributing/debugging/debugging_firefox_with_rr.rst

@@ -86,6 +86,12 @@ If you want to debug a particular part of code, you can use the :code:`MOZ_DBG`
 
 You can combine that with the -M and -g flags to jump to a particular point in a particular process's lifetime.
 
+Pernosco
+--------
+
+Pernosco is a commercial service for providing 'omniscient' debugging of :code:`rr` 
+traces. If you have an trace you'd like to debug, see :doc:`debugging_firefox_with_pernosco`
+
 Get help!
 ---------