Commit Graph

52 Commits

Author SHA1 Message Date
Dustin J. Mitchell
56f55d44fc Bug 1437562: only use ..:branch:default role, not :*; r=bstack
MozReview-Commit-ID: FzZpVvYPU5a

--HG--
extra : rebase_source : c84b68800bb252cc356f8fa82214d38423faea57
2018-02-12 18:58:05 +00:00
Aki Sasaki
b926a4ca04 bug 1328719 - task.extra.cron + task.payload.env.ACTION_TASK_GROUP_ID. r=bstack r=dustin 2018-01-12 22:17:39 -08:00
Aki Sasaki
81c122ddd6 bug 1328719 - add task.extra.tasks_for to help rebuild the decision task. r=dustin
MozReview-Commit-ID: K5ABhc3AKHK

--HG--
extra : rebase_source : 387e9744488c405619adec5b18c3b31537975845
extra : source : 9182d9f9eb70a31c0777ddd7f82182ed646fd210
2018-01-03 16:16:41 -08:00
Tom Prince
d6bdc5ded0 Bug 1424138: Stop sending pulse message to treeherder staging routes; r=camd,dustin
MozReview-Commit-ID: 1F3ekltLgY0

--HG--
extra : rebase_source : 9f363df469fd9c76ae38733d65ee2494fc3abffc
2017-12-07 21:19:00 -07:00
Rail Aliiev
484847938d Bug 1420469 - action task routes use literal "$ownTaskId" r=bstack
MozReview-Commit-ID: CsCtSpvDIip

--HG--
extra : rebase_source : 822c33e424218c3601a92a056a96f2c5c0431a49
extra : source : d951d9282368ea82c8111680a713ea6ebd737426
2017-11-24 12:56:24 -05:00
Rail Aliiev
87f83a95a1 Bug 1409577 - Add static indexes to decision task r=dustin 2017-10-18 11:55:01 -04:00
Mike Hommey
9f15f7d6b2 Bug 1409260 - Remove base-repository exception in .taskcluster.yml. r=dustin,gps
Per the comments in .taskcluster.yml, the exception is there because
some jobs were still using tc-vcs, but they aren't anymore, per bug
1383973.

--HG--
extra : rebase_source : dc81fd5c0ff4299712206edd272cdc33095bdcde
2017-10-17 14:59:53 +09:00
Brian Stack
d3a8b56012 Bug 1406209 - Use .taskcluster.yml for action tasks templates r=dustin,jonasfj
MozReview-Commit-ID: 8I8lIouV6KF

--HG--
extra : rebase_source : 6961105d1a232e93cf2f73db9ca88f8b08278dc0
2017-10-09 18:39:00 -07:00
Dustin J. Mitchell
16488779cd Bug 1394883: use /builds/worker in decision images; r=wcosta
MozReview-Commit-ID: 29cnSAr3zCW

--HG--
extra : rebase_source : dab573ee114729822c5ad9bd2674062f7ee24502
2017-08-30 15:55:35 +00:00
Phil Ringnalda
2c559035f7 Backed out 4 changesets (bug 1394883) for Windows signing chain-of-trust failures
CLOSED TREE

Backed out changeset e3f42eca51c1 (bug 1394883)
Backed out changeset 081f830cf285 (bug 1394883)
Backed out changeset 9426705a05af (bug 1394883)
Backed out changeset 3a579a5054ef (bug 1394883)

MozReview-Commit-ID: 2viO8A8arHd
2017-08-31 20:02:34 -07:00
Dustin J. Mitchell
aafeb9d9fd Bug 1394883: use /builds/worker in decision images; r=wcosta
MozReview-Commit-ID: 29cnSAr3zCW

--HG--
extra : rebase_source : dab573ee114729822c5ad9bd2674062f7ee24502
2017-08-30 15:55:35 +00:00
Gregory Szorc
6bf25db065 Bug 1390700 - Use sparse checkouts for decision task; r=dustin
The updated Docker image contains robustcheckout and run-task support
for sparse checkouts, which are obvious prerequisites.

We change the cache name so sparse and non-sparse checkouts don't
use the same working directory. If we didn't do this, tasks running
from images with old Mercurial clients or without a sparse aware
robustcheckout would fail.

The effect of using a sparse checkout is that we reduce the number
of files in the checkout from ~234,000 to ~3,600. This reduces time
for a fresh checkout from several dozen seconds to under 2s.

MozReview-Commit-ID: IJz794g8ZKH

--HG--
extra : source : 9923fffd4f64a1aa9d762e6027e0e2424a19c49c
2017-08-23 12:09:00 -07:00
Gregory Szorc
a22927dbb3 Backed out changeset 9923fffd4f64 (bug 1390700) for somehow breaking chain of trust on some Windows builds
CLOSED TREE

--HG--
extra : amend_source : e039e6840e52ee3186f4c8c0ae20d75cdccdee5a
2017-08-24 21:42:18 -07:00
Gregory Szorc
d4da1b9397 Bug 1390700 - Use sparse checkouts for decision task; r=dustin
The updated Docker image contains robustcheckout and run-task support
for sparse checkouts, which are obvious prerequisites.

We change the cache name so sparse and non-sparse checkouts don't
use the same working directory. If we didn't do this, tasks running
from images with old Mercurial clients or without a sparse aware
robustcheckout would fail.

The effect of using a sparse checkout is that we reduce the number
of files in the checkout from ~234,000 to ~3,600. This reduces time
for a fresh checkout from several dozen seconds to under 2s.

MozReview-Commit-ID: IJz794g8ZKH

--HG--
extra : rebase_source : d262c8314381a136cf5cdc5c33669c8c61818d1d
2017-08-23 12:09:00 -07:00
Gregory Arndt
85a855ffff Bug 1391288 - Rebuild decision task image with new robustcheckout r=dustin,gps
MozReview-Commit-ID: J53XXqBQsJw

--HG--
extra : rebase_source : bca5f697f049a6a3f85168aaa667ae7509e60243
2017-08-17 14:32:42 -05:00
Dustin J. Mitchell
b17efe62bb Bug 1389850: use per-level decision and image workers; r=bstack
MozReview-Commit-ID: BoUuPRUgGnm

--HG--
extra : rebase_source : 93b3fa521acd08e2ef91b47e80fbba42a94bb154
2017-08-16 13:23:13 +00:00
Dustin J. Mitchell
b5bdf8d825 Bug 1328727: revert to --base-repository=https://hg.mozilla.org/mozilla-central; r=gps
MozReview-Commit-ID: BMCVrtyooKV
2017-07-25 01:26:41 +00:00
Dustin J. Mitchell
b910cd6125 Bug 1328727: use json-e for cron decision tasks too; r=aki,jonasfj
Changes to cron decision tasks:
 - drops some unnecessary routes
 - drops tags.createdForUser: nobody@..
 - more use of environment variables within the command line

MozReview-Commit-ID: 9zoqFvwrBRs

--HG--
extra : rebase_source : 77e17089b67e5d72360d5ecb617887b7a71457fe
2017-07-21 18:08:06 +00:00
Dustin J. Mitchell
268e98b0cb Bug 1328727: use a version-1 .taskcluster.yml; r=aki,jonasfj
This makes a slight change to the resulting decision task, by putting the
comment in an environment variable. This avoids the need to perform shell
quoting (which mozilla-taskcluster was doing a poor job of anyway).

MozReview-Commit-ID: 4MRhJKDIZkk

--HG--
extra : rebase_source : 7fd4a55c574d7542825a9a2a6f37aba0b4fbec1d
2017-07-21 16:05:52 +00:00
Brian Stack
6b7f0bdd44 Bug 1380391 - Index decision tasks under pushlog_id r=dustin
MozReview-Commit-ID: 9q3iHqJ0OSb

--HG--
extra : rebase_source : 231cad344c879636b270ceef20638712c3af4134
2017-07-12 10:56:02 -07:00
Brian Stack
bce402d182 Bug 591688 - Notify when decision task fails r=dustin,gps
This uses the email provided by mozilla-taskcluster to find who to
email about failed decision tasks. It also adds some validation of
the try syntax that we've previously ignored.

Any platforms or build types specified in try sytax that don't exist
in the full task graph will throw an error.


MozReview-Commit-ID: JOKkLle7hEe

--HG--
extra : rebase_source : 65299c95c60bcb3c6a295b1f8f2f18477c2852db
2017-04-12 21:01:46 -07:00
Aki Sasaki
171f7116e8 bug 1350045 - add warning to decision docker image line. r=philor 2017-03-25 12:10:32 -07:00
Dustin J. Mitchell
5b87de297b Bug 1350045: rebuild the decision image to use the newest run-task; r=jonasfj
MozReview-Commit-ID: DeRoLfDjquc

--HG--
extra : rebase_source : 9a1f4ff66d784be38dee0c8483e239856c0fe0f4
2017-03-23 19:48:32 +00:00
Dustin J. Mitchell
05dcbf1e9f Bug 1334810: temporarily revert {{owner}}; r=garndt
MozReview-Commit-ID: FhwatSpSxDe

--HG--
extra : rebase_source : f6583854df2bfe20c1b351752a78d3c76ac040e0
2017-02-07 16:18:19 +00:00
Dustin J. Mitchell
b8d8f8a712 Bug 1252948: support for periodic taskgraphs; r=Callek,jonasfj,kmoir
This adds `.cron.yml` and a new mach command to interpret it.  While
functionality is limited to nightlies right now, there is room to expand to
more diverse periodic tasks.  Let your imagination run wild!

MozReview-Commit-ID: KxQkaUbsjQs

--HG--
extra : rebase_source : ddf0a1eadae5a1169c0ead7bcb7b9ce61b255fbf
2017-01-18 19:45:53 +00:00
Dustin J. Mitchell
17794b1e7d Bug 1252948: remove unused --revision-hash; r=garndt
MozReview-Commit-ID: CN6RNDaERJ2

--HG--
extra : rebase_source : 6c22b989813266a5014db61c9982ce682b70fbb7
2016-12-21 20:54:17 +00:00
Dustin J. Mitchell
a9383c1d2c Bug 1269443: remove now-unnecessary scopes; r=bstack
MozReview-Commit-ID: Bbk1MPKZIPW

--HG--
extra : rebase_source : 88a9e6e432262dffda698961b250f840fe8cfc19
2016-09-27 15:57:51 +00:00
Jonas Finnemann Jensen
127e057f26 Bug 1324413 - Lock image content hash for decision task; r=dustin
This will make harder to falsify a decision task.
Notably our validation code only needs to verify that the definition of the
decision task as given here matches what is used in the task definition in
the Chain-Of-Trust artifact, in order to prove that the decision task is
a result of what ran in the tree.

MozReview-Commit-ID: 4SRO7G1nyyL

--HG--
extra : rebase_source : a3b062c5adfa3c2c96a220adf9bd5d2f50d294aa
2016-12-15 17:16:14 -08:00
Jonas Finnemann Jensen
2cd49e189e Bug 1316183 - Compress docker images with zstd. r=dustin
* Compress docker images with zstd
 * Removed need for context.tar from decision task
 * Index images by level rather than project

MozReview-Commit-ID: 4RL4QXNWmpd

--HG--
extra : rebase_source : 677d8030a15af3288866a70fc648a10b22c396a3
2016-11-07 11:26:27 -08:00
Gregory Szorc
1deb8dd0e5 Bug 1311791 - Explicitly declare path to shared hg store; r=dustin
332a08725ed0 changed the store path logic in a quick and crude manner.
The code could lead to multiple shared stores existing on a cache
if checkouts were in different parent directories.

This commit refactors the code to explicitly declare a path to the
shared hg store. This restores the behavior of ensuring there is only
a single shared store per cache.

MozReview-Commit-ID: 19Aa1QVrVQ8

--HG--
extra : rebase_source : f74c0ab9cba2bf18e335f24c86dd095335961723
2016-10-20 12:56:43 -07:00
Aki Sasaki
608368610f bug 1298553 - enable chain of trust generation in the decision task. r=dustin
MozReview-Commit-ID: 7lmGt8weVgd

--HG--
extra : rebase_source : 231ff3fa94ef245399cd251b4684d2c8b1379642
2016-09-27 12:45:22 -07:00
Chris Manchester
2c2a464982 Bug 1252235 - Make try syntax available to mozharness from TaskCluster through an environment variable. r=armenzg
MozReview-Commit-ID: HIVIXPQXnQs

--HG--
extra : rebase_source : aa6ed39fdaa5bec568fabc562319f3a2ed4f4c47
2016-08-29 16:33:24 -07:00
Gregory Szorc
ef92f5ddf5 Bug 1298947 - Tag and use decision image 0.1.6; r=dustin
Containing the Mercurial 3.9.1 version bump and the change to pin the
hg.mo fingerprint from a TC secret.

MozReview-Commit-ID: LVU7P0LqIvD

--HG--
extra : rebase_source : 8ad46e014ba9840c9972b51ea43e8ccf14492cf2
2016-09-01 15:21:02 -07:00
Gregory Szorc
e1f9b3132b Bug 1298455 - Tag and use decision image 0.1.5; r=dustin
MozReview-Commit-ID: Fmc4l70onV2

--HG--
extra : rebase_source : 08a34e48faa44a25a52273a35b835c71ef178c15
2016-08-26 11:10:30 -07:00
Dustin J. Mitchell
f7a9bbac26 Bug 1293733: accept pushdate from command line; r=garndt
MozReview-Commit-ID: BrGiowlMVCa

--HG--
extra : rebase_source : e47b9e18b9dbc9e617f9bad165318a74c4e96e36
extra : source : fc1ed3fd8584161ae4693bd1c956e8254368b6bc
2016-07-13 18:50:50 +00:00
Gregory Szorc
a00feedd9f Backed out changeset fc1ed3fd8584 (bug 1291473) on suspicion of breaking decision tasks 2016-08-08 13:15:37 -07:00
Dustin J. Mitchell
05e1597215 Bug 1291473: accept pushdate from command line; r=garndt
MozReview-Commit-ID: BrGiowlMVCa

--HG--
extra : rebase_source : cf257129fcb1f6214949407b6a3956fd9698aa5b
2016-07-13 18:50:50 +00:00
Gregory Szorc
08d735f41f Bug 1291070 - Tag and use decision image 0.1.4; r=dustin
To pull in run-task exit code fix.

MozReview-Commit-ID: 5QvcQijTRKH

--HG--
extra : rebase_source : cac011e2835f18e620293e8c05a8f49da472d925
2016-08-01 18:09:09 -07:00
Dustin J. Mitchell
ac6f7946d3 Bug 1275034: add comment about taskGroupId = taskId; r=garndt
MozReview-Commit-ID: D0vfTFpwh9u

--HG--
extra : rebase_source : a3dfdbdb9fd6ed819e86003f02f093426f371631
2016-07-20 16:54:43 +00:00
Armen Zambrano Gasparnian
6415eb5945 Bug 1288742 - Increase expiration date of all gecko decision artifacts bar image context tar balls. r=dustin
Until now Gecko decision task artifacts would expire after a week. Unfortunately, some of these artifacts are
needed in order for Treeherder to determine which tasks it can schedule on a given push.

In this change, we're changing the expiration of all artifacts except those that are related to docker image
optimizations, which, are very large files and we only want to keep for a week.

MozReview-Commit-ID: 96ahyWmoI7W

--HG--
extra : rebase_source : d0f4d65e3b5bfae832a5aaed60c0dfd58f0bdc62
2016-07-27 10:51:38 -04:00
Gregory Szorc
66dab0495d Bug 1290620 - Use run-task from decision task; r=dustin
This commit does a lot. But it's really not too difficult to comprehend
once you focus on the final state, which is basically the same as the
"lint" image and derived tasks.

Before, the "decision" image contained a "checkout-gecko" script and
"run-action" and "run-decision" scripts. The latter 2 invoked the first
script.

The "run-task" script basically does what the combination of these
scripts were doing before. So we switch to it.

While we're here, we also replaced the custom Mercurial installation in
this image with the shared install-mercurial.sh script. The
system-setup.sh script for the decision image is now short and sweet.

The YAML files for tasks using this image have been updated to use
"run-task." We no longer have to pass an environment variable to hold
command arguments. So we revert to putting these arguments inline in
the task's command. Dustin never liked passing the arguments as
environment variables, so it should make him happy ;) We add
--log-no-times because "run-task" prefixes its own timestamps on output
lines.

The path to the Gecko checkout has been changed from
/home/worker/workspace to /home/worker/checkouts to match changes made
in bug 1289643.

Finally, since "checkout-gecko-and-run" is no longer used, we delete it.

The Docker image version has been bumped accordingly.

MozReview-Commit-ID: HAGuCz6668Q

--HG--
extra : rebase_source : 9586873c21d67f8ce869b7ceee221e8fa8775539
2016-07-29 21:03:48 -07:00
Gregory Szorc
a91e4e11c1 Bug 1290516 - Remove {{project}} from cache path for decision and action tasks; r=dustin
Currently, the gecko-decision and action tasks maintain a separate cache
for each "project" (Firefox repo) source checkout. In my opinion, this
is not necessary.

Separate caches/source checkouts mean a lower chance for a cache hit:
a lower chance that any given worker will have a source checkout readily
available for the "project" being built. They also mean workers store
more data and files in caches.

By using a shared cache for source checkouts, workers will ideally
only have a single source checkout cache and will be pretty much
guaranteed to have a checkout readily available (the main exception
being the first task on a freshly-provisioned worker).

A downside of a shared cache is that workers may spend more time doing
`hg update` operations. e.g. if the last task was from mozilla-release
and the current task is from mozilla-central, thousands of files will
be updated. However, I believe we will still be better off because
separate caches *increase* the probability for a full checkout, which
involves writing ~150,000 files! So I believe this change will lower
the overall amount of VCS work done by workers while possibly
sacrificing the lowest possible execution time (which is achieved by
minimizing the `hg update` work).

It's worth noting that `hg robustcheckout` (which is used by these
tasks) does the equivalent of an `hg purge --all`. This ensures that all
untracked and ignored files from previous tasks are deleted. So there
should be no contamination from previous tasks.

MozReview-Commit-ID: I5Ng3xFEfou

--HG--
extra : rebase_source : db423e5005d1e26d7dfebbbfa28893fc32caef6f
2016-07-29 10:09:06 -07:00
Gregory Szorc
df70fc2d86 Bug 1247168 - Make mozilla-unified the base repository for Firefox; r=dustin
https://hg.mozilla.org/mozilla-unified contains heads from all the
major Firefox repos. In addition, it is encoded on the server in such a
way that it is several hundred megabytes smaller despite containing
30,000+ more changesets. This means faster clones and faster
operations.

Cloning this repo does require a new version of Mercurial. So if any
TC tasks are using an ancient Mercurial - one that is vulnerable to
known CVE issues in fact - this will flush them out.

MozReview-Commit-ID: 2VHDa6FEeeJ

--HG--
extra : rebase_source : db4990bcde0503fd14d82a5d16c71adbb4f92be3
extra : histedit_source : 6c555bd3df12536d1c48b45d8cc76611e3f7032f
2016-07-19 16:46:25 -07:00
Gregory Szorc
6aa899bbf4 Bug 1247168 - Tag and use version 0.1.2 of the decision image; r=dustin
Changes to the decision Docker image have been compelted. We're ready to
use the new image.

We tag the image, update version references, change the task caches
so the new Mercurial pooled storage from the robustcheckout extension is
used, and convert the decision tasks to run as the "worker" user.

MozReview-Commit-ID: 61v9Ivy59zG

--HG--
extra : rebase_source : 640318a87660950c5e0680867a1bfdd68e35f127
extra : histedit_source : ec53fc576c00e5f2053167b37544ac7afccaecb5
2016-07-21 11:32:07 -07:00
Gregory Szorc
94d4854313 Bug 1247168 - Actually use workspace cache in decision task; r=dustin
The decision task configures a /home/worker/workspace cache. However,
the command we run in the container references a "workspace" relative
path. From logs in automation, it appears that PWD during execution
is "/" because "workspace" is being resolved to "/workspace." The
net result of this is we appear to be performing a VCS clone+checkout
on every single task.

This commit fixes the paths so our workspace cache is actually used.

MozReview-Commit-ID: Kj6REep5bSs

--HG--
extra : rebase_source : 8cd8be43dfd34f2970b47721c3da8e3957a8bfed
extra : histedit_source : b81dce523a88e44eb3fa8b1a68840066edca382d
2016-07-20 15:34:10 -07:00
Dustin J. Mitchell
fa358ba635 Bug 1273645: remove unnecessary scope from decision task; r=garndt
MozReview-Commit-ID: ViGVLPCXc6

--HG--
extra : rebase_source : d7c5910f9789f0e68b62fe4d1d5c06b9fdda1f76
2016-07-11 22:07:12 +00:00
Gregory Arndt
6e0a99decc Bug 1282864 - Remove unecessary revision hash info from decision task r=dustin
MozReview-Commit-ID: ICbOpbVwEhv

--HG--
extra : rebase_source : 8146b78b9c1d7c650cceb31ff6484c53d3f4f84b
2016-06-28 13:02:04 -05:00
Gregory Arndt
54542098dc Bug 1280129 - Update tasks to use v2 treeherder routes r=dustin
Jobs reporting to treeherder should rely on the task route for project,
revision, and pushlog ID rather than things stuffed into task.extra.treeherder.

This also removes the need for a revision_hash that was calculated by mozilla-taskcluster.

MozReview-Commit-ID: EcQM9QRZzgG

--HG--
extra : rebase_source : f04f6724feef2dd51b4b98c67c9a261b093f452b
extra : amend_source : 0590605834d93359206f49edd94396c43b57f6dd
2016-06-14 12:56:25 -05:00
Dustin J. Mitchell
76cf8381c4 Bug 1274385: don't retry decision tasks; r=garndt
MozReview-Commit-ID: IjnhKWgoxAG

--HG--
extra : rebase_source : 61e41dff628c1dce1907a85b3555ff9c3c1851de
2016-05-19 20:09:29 +00:00
Dustin J. Mitchell
19607e0f91 Bug 1274717: use a level-appropriate scope; r=garndt a=KWierso
MozReview-Commit-ID: KU7D06KauBK

--HG--
extra : source : 2b520d36529bcc2ca820bec79e55a4016469b3c7
extra : amend_source : 33cd5e37bf13fb307729966fba75b1fd0810485f
2016-05-20 23:00:15 +00:00