gecko-dev/.taskcluster.yml
Gregory Szorc a91e4e11c1 Bug 1290516 - Remove {{project}} from cache path for decision and action tasks; r=dustin
Currently, the gecko-decision and action tasks maintain a separate cache
for each "project" (Firefox repo) source checkout. In my opinion, this
is not necessary.

Separate caches/source checkouts mean a lower chance for a cache hit:
a lower chance that any given worker will have a source checkout readily
available for the "project" being built. They also mean workers store
more data and files in caches.

By using a shared cache for source checkouts, workers will ideally
only have a single source checkout cache and will be pretty much
guaranteed to have a checkout readily available (the main exception
being the first task on a freshly-provisioned worker).

A downside of a shared cache is that workers may spend more time doing
`hg update` operations. e.g. if the last task was from mozilla-release
and the current task is from mozilla-central, thousands of files will
be updated. However, I believe we will still be better off because
separate caches *increase* the probability for a full checkout, which
involves writing ~150,000 files! So I believe this change will lower
the overall amount of VCS work done by workers while possibly
sacrificing the lowest possible execution time (which is achieved by
minimizing the `hg update` work).

It's worth noting that `hg robustcheckout` (which is used by these
tasks) does the equivalent of an `hg purge --all`. This ensures that all
untracked and ignored files from previous tasks are deleted. So there
should be no contamination from previous tasks.

MozReview-Commit-ID: I5Ng3xFEfou

--HG--
extra : rebase_source : db423e5005d1e26d7dfebbbfa28893fc32caef6f
2016-07-29 10:09:06 -07:00

113 lines
3.8 KiB
YAML

---
version: 0
metadata:
name: 'Taskcluster tasks for Gecko'
description: "The taskcluster task graph for Gecko trees"
owner: mozilla-taskcluster-maintenance@mozilla.com
source: {{{source}}}
scopes:
# Note the below scopes are insecure however these get overriden on the server
# side to whatever scopes are set by mozilla-taskcluster.
- queue:*
- docker-worker:*
- scheduler:*
# Available mustache parameters (see the mozilla-taskcluster source):
#
# - owner: push user (email address)
# - source: URL of this YAML file
# - url: repository URL
# - project: alias for the destination repository (basename of
# the repo url)
# - level: SCM level of the destination repository
# (1 = try, 3 = core)
# - revision: (short) hg revision of the head of the push
# - revision_hash: (long) hg revision of the head of the push
# - comment: comment of the push
# - pushlog_id: id in the pushlog table of the repository
#
# and functions:
# - as_slugid: convert a label into a slugId
# - from_now: generate a timestamp at a fixed offset from now
tasks:
- taskId: '{{#as_slugid}}decision task{{/as_slugid}}'
task:
created: '{{now}}'
deadline: '{{#from_now}}1 day{{/from_now}}'
expires: '{{#from_now}}14 day{{/from_now}}'
metadata:
owner: mozilla-taskcluster-maintenance@mozilla.com
source: {{{source}}}
name: "Gecko Decision Task"
description: |
The task that creates all of the other tasks in the task graph
workerType: "gecko-decision"
provisionerId: "aws-provisioner-v1"
tags:
createdForUser: {{owner}}
scopes:
# Bug 1269443: cache scopes, etc. must be listed explicitly
- "docker-worker:cache:level-{{level}}-*"
- "docker-worker:cache:tooltool-cache"
# mozilla-taskcluster will append the appropriate assume:repo:<repo>
# scope here.
routes:
- "index.gecko.v2.{{project}}.latest.firefox.decision"
- "tc-treeherder.v2.{{project}}.{{revision}}.{{pushlog_id}}"
- "tc-treeherder-stage.v2.{{project}}.{{revision}}.{{pushlog_id}}"
payload:
env:
# checkout-gecko uses these to check out the source; the inputs
# to `mach taskgraph decision` are all on the command line.
GECKO_BASE_REPOSITORY: 'https://hg.mozilla.org/mozilla-unified'
GECKO_HEAD_REPOSITORY: '{{{url}}}'
GECKO_HEAD_REF: '{{revision}}'
GECKO_HEAD_REV: '{{revision}}'
# Arguments passed into `mach taskgraph decision`
# TODO use mozilla-unified for the base repository once the tc-vcs
# tar.gz archives are created or tc-vcs isn't being used.
DECISION_ARGS: >
--pushlog-id='{{pushlog_id}}'
--project='{{project}}'
--message='{{comment}}'
--owner='{{owner}}'
--level='{{level}}'
--base-repository='https://hg.mozilla.org/mozilla-central'
--head-repository='{{{url}}}'
--head-ref='{{revision}}'
--head-rev='{{revision}}'
--revision-hash='{{revision_hash}}'
cache:
level-{{level}}-hg-shared: /home/worker/hg-shared
level-{{level}}-gecko-decision: /home/worker/workspace
features:
taskclusterProxy: true
# Note: This task is built server side without the context or tooling that
# exist in tree so we must hard code the version
image: 'taskcluster/decision:0.1.2'
maxRunTime: 1800
command:
- /home/worker/bin/run-decision
artifacts:
'public':
type: 'directory'
path: '/home/worker/artifacts'
expires: '{{#from_now}}7 days{{/from_now}}'
extra:
treeherder:
symbol: D