Bug 1560755 - Merge Android PGO profile data in run task; r=firefox-build-system-reviewers,chmanchester

Android profile runs don't always fully write out the profile data. In
this case, the corrupted profile data is successfully uploaded, but
future profile-use PGO builds try to merge the data and fail. Retrying
the profile-use builds doesn't help, since they all pull from the same
job that published the corrupt data.

We can detect this in the run task by using llvm-profdata merge, and if
the merge fails the task can automatically be retried. Note that the
data gets redundantly merged in the profile-use build, but it may not be
possible to run the merge in the run task on all platforms (eg: OSX), so
we have to keep the merge there as well.

Differential Revision: https://phabricator.services.mozilla.com/D36294

--HG--
extra : moz-landing-system : lando

taskcluster/ci/generate-profile/kind.yml

@@ -5,12 +5,14 @@
 loader: taskgraph.loader.transform:loader
 
 kind-dependencies:
+    - toolchain
     - instrumented-build
 
 transforms:
     - taskgraph.transforms.build_attrs:transforms
     - taskgraph.transforms.release_deps:transforms
     - taskgraph.transforms.run_pgo_profile:transforms
+    - taskgraph.transforms.use_toolchains:transforms
     - taskgraph.transforms.job:transforms
     - taskgraph.transforms.task:transforms
 
@@ -125,6 +127,8 @@ jobs:
                 - android/android_common.py
                 - android/androidarm_4_3.py
                 - android/android_pgo.py
+        toolchains:
+            - linux64-clang
 
     win32-shippable/opt:
         description: "Win32 Profile Generation"

testing/mozharness/scripts/android_emulator_pgo.py

@@ -277,13 +277,29 @@ class AndroidProfileRun(TestingMixin, BaseScript, MozbaseMixin,
             self.fatal('INFRA-ERROR: Failed with an ADBTimeoutError',
                        EXIT_STATUS_DICT[TBPL_RETRY])
 
+        # We normally merge as part of a GENERATED_FILES step in the profile-use
+        # build, but Android runs sometimes result in a truncated profile. We do
+        # a merge here to make sure the data isn't corrupt so we can retry the
+        # 'run' task if necessary.
+        merge_cmd = [
+            '/builds/worker/workspace/build/clang/bin/llvm-profdata',
+            'merge',
+            '/builds/worker/workspace/default.profraw',
+            '-o',
+            '/builds/worker/workspace/merged.profraw',
+        ]
+        rc = subprocess.call(merge_cmd)
+        if rc != 0:
+            self.fatal('INFRA-ERROR: Failed to merge profile data. Corrupt profile?',
+                       EXIT_STATUS_DICT[TBPL_RETRY])
+
         # tarfile doesn't support xz in this version of Python
         tar_cmd = [
             'tar',
             '-acvf',
             '/builds/worker/artifacts/profdata.tar.xz',
             '-C', '/builds/worker/workspace',
-            'default.profraw',
+            'merged.profraw',
             'en-US.log',
         ]
         subprocess.check_call(tar_cmd)