Bug 1410969 - TaskCluster task for generating JSON files with Bugzilla metadata; r=ahal

We now have a "source" task that produces JSON files with per-file
Bugzilla components and a list of files missing a declared Bugzilla
component. gzip variations are also produced.

The files are published in the index so clients can query e.g.
gecko.v2.mozilla-central.latest.source.source-bugzilla-info/public/components.json
and get the latest metadata. This should help alleviate the need for
querying the moz.build evaluation API on hg.mozilla.org - or at least
facilitate bulk queries of the data from a static source.

MozReview-Commit-ID: 9fAoPSt4bxq

--HG--
extra : rebase_source : bad6912a5e2bf5f4791e97f0d0b2572d70007e9a

taskcluster/ci/source-test/file-metadata.yml

@@ -0,0 +1,33 @@
+job-defaults:
+    platform: lint/opt
+    treeherder:
+        kind: test
+        tier: 1
+    worker-type: aws-provisioner-v1/gecko-t-linux-xlarge
+    worker:
+        docker-image: {in-tree: "lint"}
+        max-run-time: 1800
+
+bugzilla-components:
+    description: Generate metadata about source files and Bugzilla
+    treeherder:
+        symbol: Bugzilla
+    index:
+        product: source
+        job-name: source-bugzilla-info
+    run:
+        using: run-task
+        command: >
+            mkdir -p /builds/worker/artifacts
+            && cd /builds/worker/checkouts/gecko
+            && ./mach file-info bugzilla-component --format json '**' > /builds/worker/artifacts/components.json
+            && ./mach file-info missing-bugzilla --format json '**' | tee /builds/worker/artifacts/missing.json
+            && gzip --keep /builds/worker/artifacts/*.json
+    worker:
+        artifacts:
+            - type: directory
+              path: /builds/worker/artifacts
+              name: public
+    when:
+        files-changed:
+            - '**'

taskcluster/ci/source-test/kind.yml

@@ -13,6 +13,7 @@ transforms:
 jobs-from:
    - cram.yml
    - doc.yml
+   - file-metadata.yml
    - mocha.yml
    - mozlint.yml
    - python.yml

taskcluster/taskgraph/transforms/gecko_v2_whitelist.py

@@ -95,6 +95,7 @@ JOB_NAME_WHITELIST = set([
     'sm-rootanalysis-debug',
     'sm-rust-bindings-debug',
     'sm-tsan-opt',
+    'source-bugzilla-info',
     'win32-add-on-devel',
     'win32-clang-tidy',
     'win32-debug',

taskcluster/taskgraph/transforms/task.py

@@ -113,7 +113,13 @@ task_description_schema = Schema({
     # if omitted, the build will not be indexed.
     Optional('index'): {
         # the name of the product this build produces
-        'product': Any('firefox', 'mobile', 'static-analysis', 'devedition'),
+        'product': Any(
+            'firefox',
+            'mobile',
+            'static-analysis',
+            'devedition',
+            'source',
+        ),
 
         # the names to use for this job in the TaskCluster index
         'job-name': basestring,