Synchronized with hgext/robustcheckout/__init.py
from https://hg.mozilla.org/hgcustom/version-control-tools at revision
d2140637eaf3f91fefa7c2f44cbaabf4c19faeb3.
MozReview-Commit-ID: 676o5IVE536
--HG--
extra : rebase_source : 8292cceb465d295f50db68ee0e0fdfb6f6797c26
We need to be able to run the copydirs logic in desktop_unittest.py independently
of the 'run-tests' action to enable a smooth interactive debugging workflow.
MozReview-Commit-ID: ATftu2NnhQq
--HG--
extra : rebase_source : 915ddfad28b4660af1066fc3279ca3d11164a4b6
With bug 1276895 our custom code to unzip test archives has been removed. But we still need it
because our Windows slave nodes in Jenkins do not have the unzip command installed. This code
can finally removed once bug 1258539 is fixed.
MozReview-Commit-ID: 4WbFrQ524l1
--HG--
extra : rebase_source : 94d7c782285827bfaf347a1e44a6ce952aec6978
There are two subsets of functional tests which use local or remote test data.
The appropriate subset can be selected via the --tag option which comes from Marionette.
MozReview-Commit-ID: Bfu6IsXVc2T
--HG--
extra : rebase_source : 7b09a0bc586277210647993c3563d71330af63d1
There are two subsets of functional tests which use local or remote test data.
The appropriate subset can be selected via the --tag option which comes from Marionette.
MozReview-Commit-ID: Bfu6IsXVc2T
--HG--
extra : rebase_source : 43069f038bfb1427253684af896bcacb2cb6992c
This adds the ability to use the command line flag '--jscov-dir-prefix' to collect javascript code coverage from xpcshell tests and output it into the specified directory as a JSON file.
MozReview-Commit-ID: 3MZm73SNChL
--HG--
extra : transplant_source : c%9B%DE%A93w%E7%11%89%BE-%E8%D9%18%BC%12z%0A%0E%E4
This makes sure mozharness will always extract the mach binary and it will
be available when debugging interactive loaners.
MozReview-Commit-ID: BIXcCm4LzE2
--HG--
extra : rebase_source : 66cef916f7e86115b4f002ebad2550b70c0ae141
MozReview-Commit-ID: KFlaQoeGPRc
* I guess this has to be uplifted to aurora and included in the aurora->beta
merge
--HG--
extra : rebase_source : eb97b061f193e0f6d976ea374a3ed2677435520b
extra : amend_source : 843c4791f09dc39e21499dd1782614b2c5442b7e
this patch:
* forces always passing '-r' when pushing after a migration run
* removes '--new-branch' from beta_to_release push. we don't generate new branches on releases anymore because of release promotion!
* has migration runs use a unique share dir for each repo.
this can be used as a stop gap. maybe even a more permanent solution so we don't need to add complexity to robustcheckout for an edgecase like this.
MozReview-Commit-ID: HXY5vDI1pIt
--HG--
extra : rebase_source : 717e48dbf79115817e48adba8ed2f082d832fca0
This commit teaches the resource monitor in mozharness to emit
Perfherder data for system metrics and step times. This will
allow us to see when the timing or resource characteristics
of jobs in automation changes.
The recorded data includes overall CPU percent usage and I/O.
Each step has its time and CPU percent recorded. There is
certainly more data we could record. However, the immediate
goal of this change is to see if the data provides any benefit.
I'd rather start small and expand reporting once value from
this data is proved.
The wonkiest part of this patch is likely the mechanism to
define the Perfherder "test" names. We don't appear to have
an identifier in mozharness suitable for distinguishing
between job types. e.g. the "desktop_unittest.py" script is
responsible for running a few dozen jobs. So we invent code
for creating an identifier from the script config options.
I /think/ Treeherder will automatically assign the
project/branch, platform, and build type, which is why these
aren't included in the identifier.
MozReview-Commit-ID: HjhtXfxOvzJ
--HG--
extra : rebase_source : a3f0f2de4a091cde10c5a6815f1b4646bb5dc2f2
Having the latest schema available seems like a good thing. This is a
direct copy of schemas/performance-artifact.json from
https://github.com/mozilla/treeherder.git at commit
7bed1b22ceb01e3e71536fa1c4ecd14ddc87e803.
MozReview-Commit-ID: JQC4CeW6szM
--HG--
extra : rebase_source : a57d4e93b9334b5c571b05e0ef52f637a45432dd
Currently, only Talos accesses this file. An uncoming commit will add
a non-Talos consumer. Enable all mozharness consumers to access the
file by including it in the mozharness directory (previously it was
part of the Talos test archive).
MozReview-Commit-ID: ADlCj9E5BwC
--HG--
rename : testing/talos/treeherder-schemas/performance-artifact.json => testing/mozharness/external_tools/performance-artifact-schema.json
extra : rebase_source : ce5fcaf700941ce260c97c6daeefa07b4ef5e617
robustcheckout barfs on symbolic revisions when using "revision."
MozReview-Commit-ID: B7YXqbWG0G1
--HG--
extra : rebase_source : d852930ac24be79004bce978c8ed6542ab58600f
MozReview-Commit-ID: GD3vspSFmTa
* fix clean_repos. it expects vcs_config key revision but now we use branch
--HG--
extra : rebase_source : d09622ca30eb1c7face42892f149812e5ae5a26a
extra : amend_source : cf7491d93ff2ea9ca13929606368fd443c8f026b
These all definitely use the modern HgVCS because it is explicitly
specified in the configs. So without this change, these configs would
fail since --revision rejects symbolic names.
MozReview-Commit-ID: 2SlVWNVwc08
--HG--
extra : rebase_source : 5e3d0cd075b5a35c4ad0d95b9ec0d6b3715d5080
I'm pretty sure partner repacks are using the modern HgVCS and not
HgToolVCS. So have them use "branch" instead of "revision" for symbolic
revisions.
MozReview-Commit-ID: BuEHGFmK6cK
--HG--
extra : rebase_source : 149d31434a2cf84ff7ade8bff9e7abe4e15e3758
I /think/ hazard builds are currently only performed on TC, which doesn't
use the VCS settings in mozharness. So I don't think this could possibly
break automation.
While I was here, I removed a reference to hgtool since we're no longer
using hgtool in this job.
MozReview-Commit-ID: fQj2MzpGRT
--HG--
extra : rebase_source : f0d0880a50c0597b10c6e97c13f04ae7cf6cc131
The normal MercurialVCS now supports querying pushlog. Use it.
This isn't really relevant to the bug summary and other commits in this
series. But I already wrote this commit and was too lazy to create a
new bug for it.
MozReview-Commit-ID: C97Zgox3xKB
--HG--
extra : rebase_source : e84b6e723e1d481a24a4ba0812735d8b34acd218
We remove the old config settings related to hgtool and switch
the "revision" option to "branch" because it defines a symbolic
revision.
MozReview-Commit-ID: Eq4R5a2tv2V
--HG--
extra : rebase_source : 4d85abbc2db6f499206d741f98c316b9c521b4ee
This is a regression from cd58054bde90 (bug 1274693).
MozReview-Commit-ID: DoyeGf1Cu7l
--HG--
extra : rebase_source : 3585f3a742571b26d681b1b1d102289d705780c5
Recent changes to mozharness in bug 1270317 started using pooled shared
storage for Mercurial repos. This means the "default" path in Mercurial
repos is variable depending on which repo was the first to be built on
a machine.
By default, the Firefox build system resolves the source repository
from `hg paths default`. This is now incorrect default behavior in
automation.
We fix the regression by setting MOZ_SOURCE_REPO in the environment to
path to the repository that mozharness is currently building.
MozReview-Commit-ID: 34IPf7PJfuA
--HG--
extra : rebase_source : d0d717847f1e52444ecd53ddf16716bad42809eb
extra : source : a2a4c06d9736850782d8cc52802e0207ca1bf27a
Older versions of Python 2.7 have a bug where subprocess and execve
don't like unicode in the environment variable dict. This patch
ensures we are using a str.
Pusing on a CLOSED TREE like a boss.
--HG--
extra : amend_source : 5f464f6ffd58711d31129e6d8971a85b5b75b1c7
Recent changes to mozharness in bug 1270317 started using pooled shared
storage for Mercurial repos. This means the "default" path in Mercurial
repos is variable depending on which repo was the first to be built on
a machine.
By default, the Firefox build system resolves the source repository
from `hg paths default`. This is now incorrect default behavior in
automation.
We fix the regression by setting MOZ_SOURCE_REPO in the environment to
path to the repository that mozharness is currently building.
MozReview-Commit-ID: 34IPf7PJfuA
--HG--
extra : rebase_source : 7b4944fc059ffc0b25a6a74072d5546baa8acefc
This is a straight import of version-control-tools:hgext/robustcheckout/__init__.py
at d115bc9578dac4ae2694ea93bb9662de9e6b0b67. Code was reviewed in bug 1274095.
Should be a rubber stamp review.
MozReview-Commit-ID: CEXR9uCHBfI
--HG--
extra : rebase_source : c71bf0bc965993183a197e93459826137392d9e7
In bug 1262260 and 1250904, we are building some tools to make debugging tests in
interactive jobs easier. Part of this includes making it easier to run mozharness
scripts with various configurations. Some of these tools depend on a "run-tests"
action being present, but in marionette.py, this action is called "run-marionette"
instead.
Rather than make these tools special case marionette, let's just make marionette.py
consistent with every other mozharness script.
MozReview-Commit-ID: CrPgjUkCBQR
--HG--
extra : rebase_source : 0ab6cc8d5cfcc329b77a9e3096b9d377edfe3c40
After bug 1270317 we require the use of shared repos in automation. We
refuse to operate if the vcs_share_base config option or corresponding
environment variable isn't defined (at least with the "hg" VCS tool).
It looks like the multi_locale configs are using the "hg" tool without
a shared directory defined. So define the shared directory to appease
the "hg" tool.
Pushing on a CLOSED TREE.
MozReview-Commit-ID: 9TLrTYaMQzT
--HG--
extra : amend_source : 0c47f3615c4c91454e6076c3c3e864604678d1ea
I haven't tested this. Fortunately B2G isn't a supported platform
any more, so I'm pretty sure I don't have to care about this working.
MozReview-Commit-ID: 2B1HsvYXtsX
--HG--
extra : rebase_source : 6fd9f72bd74485ad8f189024d6d04065567efcd0
extra : amend_source : e522c9f47e0bd4c4469e6e4c3adefbd7faecb36d
The new tool has different behavior for "revision" and "branch:" if
you pass a name to "revision" and it already resolves, it won't
attempt a pull because it thinks it already has the revision. That
would be bad.
So convert consumers using the "hg" tool to pass "branch" when
using symbol names.
MozReview-Commit-ID: Ix2jt3XabW2
--HG--
extra : rebase_source : 0c1dc950b3ab1e0b5ce4c449ed2056ac2b65cbf9
extra : amend_source : c522ba6dca22387bbcf77f92d94957cba3b4f3c7
hgtool printed the hg version info when running. This is useful data
when debugging Mercurial failures. Add it back in.
We also add `hg debuginstall`, which prints useful bits about the
install, including the Python path and version.
MozReview-Commit-ID: IeKhfWDXEys
--HG--
extra : rebase_source : 2cda6334353935a700373d6204f40428cb10518b
extra : histedit_source : 2fdaaf0a32525a60945686ea6ee9ae1154e6259d
Now that the MercurialVCS VCS tool does things optimally, we no longer
need to use hgtool!
Again, this will effectively require a modern Mercurial version or
things will fail.
MozReview-Commit-ID: 9SM9qfYGlU6
--HG--
extra : rebase_source : 0376250e782f03f0a375ae42cf7f9f30a93eef5b
extra : source : d01331bbdebe58edb59f222b608a2f1796e33004
extra : histedit_source : 890daa7fd8337ce6a222d768412f2a51c82d8c12
Functionality for doing an optimal clone/pull+share+purge+update is now
implemented in the robustcheckout extension so it can be implemented in one
place and used by all the various tools needing to perform a "robust"
checkout using optimal practices.
This commit switches the MercurialVCS to use it.
Functionality for interfacing with shared repos and associated tests have
been removed because this is all implemented and tested in robustcheckout.
Various other tests have also been removed because they are redundant with
tests in the robustcheckout extension.
MozReview-Commit-ID: FGvmSHKM5e0
--HG--
extra : rebase_source : 8f31a1e79d448478fa63b17582313409ac06fe69
extra : histedit_source : 3031dd8f83b0c64abc110252fd270f1917168663
MercurialVCS doesn't currently implement the VCSMixin interface.
This commit copies the implementation of query_pushinfo() from
HgtoolVCS to MercurialVCS so it implements the interface.
MozReview-Commit-ID: LKpLVhQoKww
--HG--
extra : rebase_source : 6dad5a86e6f9018ca5c3cdbd5fb37082ec700ef7
extra : histedit_source : 92daf0709d8913c1ee5db549bdf5dd453840f40b
We currently have a "clone_by_revision" property that indicates to
perform a `hg clone -r`. We use it for cloning from Try.
Cloning single revisions undermines the benefits of clone bundles. So,
I'll be replacing "clone_by_revision" with a feature that clones from
another "upstream" repo then does a `hg pull -r` on the wanted revision.
This commit starts that work by introducing a "clone_upstream_url"
property. We define it on Try. It is currently unused.
MozReview-Commit-ID: Dohs8bCTUkB
--HG--
extra : rebase_source : ab6f9a0b270b70386435a4040b55d3362b84e51c
extra : histedit_source : 055dbf85eb762deab3c05c3092cb57d4313a6957
We had a test environment running on Python 2.6 and an ancient version
of Mercurial. AFAICT we run Python 2.7 everywhere, so this environment
can be dropped.
We also upgrade to Mercurial 3.7.3, as that is what automation now runs.
MozReview-Commit-ID: 7WTyD3CUjtj
--HG--
extra : rebase_source : 28994488cc1ffbc779ac4f25ec0cbbd2749d169d
extra : histedit_source : bdd034b5c2d3cc479f58b614cf368372c81c8896
We currently have our own system monitor serialization in
building.py. It predates as_dict() from mozsystemmonitor. Let's
use the "upstream" data format so we only have a single format
to consume.
This change required updating the in-tree resource viewer to
be compatible with the new data format.
This commit stops short of getting rid of the existing
data massaging code in building.py. Another day perhaps.
MozReview-Commit-ID: 1OJrSiyJjMX
--HG--
extra : rebase_source : 9782b2164d1735ed0872fe8c1637204d5b3b1313
Data is too valuable to waste. Let's upload the raw resource
data captured by the resource monitor so we can look at resource
usage in more detail whenever we want.
MozReview-Commit-ID: 5Q2EanSMD9v
--HG--
extra : rebase_source : 596fb330c8e1acab56cc3590f6a3b28cef2ebd01
mozsystemmonitor 0.1 has been released. It features support
for retrieving a dict of gathered data, making JSON export easy.
Let's use it. This version also requires psutil>=3.1.1, so bump
that version as well.
MozReview-Commit-ID: 9DsEQNjV6kJ
--HG--
extra : rebase_source : 45681261113e2d4624fc348ed8f7335d377752fa
The system resource utilization during job execution is important: it
gives us an idea of the efficiency (or lack thereof) of activities.
As bug 1271035 showed us, there can be some really wonky things going
on during job execution. To help us notice these things, this commit
prints some overall resource utilization data with the special
"TinderboxPrint" syntax so it appears in Treeherder. This should
hopefully draw the attention of more eye balls and cause people to
ask questions about what jobs are doing.
This supplements the existing printing of total resource usage in the
logs. Unfortunately nobody was really looking at that data because it
wasn't exposed that well. This commit should change that.
MozReview-Commit-ID: AXNRDS9lrOd
--HG--
extra : rebase_source : c5e6b440092853649456d89a1f7dc370ca4ec29a
I followed the guide at:
https://wiki.mozilla.org/Mobile/Fennec/Android/Task_Cluster_notes
To identify what to change.
MozReview-Commit-ID: HnKSSqym0aA
--HG--
rename : testing/mozharness/configs/builds/releng_sub_android_configs/64_api_15_frontend.py => testing/mozharness/configs/builds/releng_sub_android_configs/64_test.py
rename : testing/taskcluster/tasks/builds/android_api_15_frontend.yml => testing/taskcluster/tasks/builds/android_test.yml
extra : rebase_source : a4080ecc8afab781cbd81de7b2d2c1f9b3968757
hgtool printed the hg version info when running. This is useful data
when debugging Mercurial failures. Add it back in.
We also add `hg debuginstall`, which prints useful bits about the
install, including the Python path and version.
MozReview-Commit-ID: IeKhfWDXEys
--HG--
extra : rebase_source : 93496db608a2f9480e521c526e30a25646583997
Now that the MercurialVCS VCS tool does things optimally, we no longer
need to use hgtool!
Again, this will effectively require a modern Mercurial version or
things will fail.
MozReview-Commit-ID: 9SM9qfYGlU6
--HG--
extra : rebase_source : 541303fb53a4ebd9aa4fd3313f8c72182d01ad37
The code for ensure_repo_and_revision() has been completely overhauled.
We now require shared repos using auto pooled storage via the share
extension. This ensures that only a single copy of a logical
repository's history is stored on disk. e.g. if you clone
mozilla-central, inbound, and try, they'll all automatically share the
same storage.
The new code ensures the destination repo is using modern conventions
and will delete the destination repo if it isn't. So once this gets
deployed to production, machines will slowly start using optimal
storage. This should make VCS operations significantly faster.
Another optimization that is now in here is we check for presence of the
wanted revision before doing `hg pull`. This saves some communication
with the server if the revision is already present locally.
This change effectively requires a modern version of Mercurial to be
installed to run ensure_repo_and_revision(). Since Mercurial <3.7.3 has
security vulnerabilities, we shouldn't be running <3.7.3 in automation.
So I think this will be OK. If not, it will certainly be easy to
identify which machines aren't updated!
MozReview-Commit-ID: 62jtAsDj7rU
--HG--
extra : rebase_source : a43d7f54b16e71940e45ddf05402449575fccfee
MercurialVCS doesn't currently implement the VCSMixin interface.
This commit copies the implementation of query_pushinfo() from
HgtoolVCS to MercurialVCS so it implements the interface.
MozReview-Commit-ID: LKpLVhQoKww
--HG--
extra : rebase_source : 358d37e29f9239c0b1c084c0251af7a94c1f526a
We currently have a "clone_by_revision" property that indicates to
perform a `hg clone -r`. We use it for cloning from Try.
Cloning single revisions undermines the benefits of clone bundles. So,
I'll be replacing "clone_by_revision" with a feature that clones from
another "upstream" repo then does a `hg pull -r` on the wanted revision.
This commit starts that work by introducing a "clone_upstream_url"
property. We define it on Try. It is currently unused.
MozReview-Commit-ID: Dohs8bCTUkB
--HG--
extra : rebase_source : 17b643f32747d494db04a2e80c4f94308b13618e
We no longer use <3.7 in automation. So drop support for <3.2. While I
was here, I also added magic variables to the extension so Mercurial
can react intelligently to version compatibility issues.
MozReview-Commit-ID: 4tAvQljasDR
--HG--
extra : rebase_source : ef2b5070014507533b5c0ec17449d62ba1bedad8
extra : source : 43c18590fcc7bbf7573647c2f225f4a82dd55730
The build/tools repo has a "purgelong" extension that is used to delete
long filenames on Windows. Without this extension, the APIs Mercurial
uses may run into path length issues and `hg purge` will fail.
This commit is a straight import of the purgelong extension
from https://hg.mozilla.org/build/tools minus the shebang line, which
isn't needed.
MozReview-Commit-ID: FIrEeWDf2Dl
--HG--
extra : rebase_source : ce34dc3dbcfbb9463022f3019f167a58cb396ef3
We had a test environment running on Python 2.6 and an ancient version
of Mercurial. AFAICT we run Python 2.7 everywhere, so this environment
can be dropped.
We also upgrade to Mercurial 3.7.3, as that is what automation now runs.
MozReview-Commit-ID: 7WTyD3CUjtj
--HG--
extra : rebase_source : 8e37b215fb2bff2f12658fd5ad3b61d631ec26c7
This will enable log parsers to find step boundaries easier.
MozReview-Commit-ID: G4OaViDd9Fv
--HG--
extra : rebase_source : a7e94e4ec088c4fed7eb2b7a1662e308296e8bb2
* forces per checkin clobber only on 'release' platforms
MozReview-Commit-ID: E7ZxdiYH47d
--HG--
extra : rebase_source : 1dc896d61f7392b4fb253f08de7b743c86f0eebc
extra : amend_source : 677fda6773448183580701718114642b405668bf
Currently, this only runs on *.java but I believe checkstyle can also
do analysis on our xml files, so we may need to update this in the future.
MozReview-Commit-ID: 4F6vSHaUZed
--HG--
extra : rebase_source : 5347b55a0cfe8abbc81806a637296b467b7c7b0e
Allow passing version and build_number via buildbot properties. This will allow
us add buildbot builders without hardcoded parameters depending on release
configs.
MozReview-Commit-ID: 3zGiCJ5z36X
--HG--
extra : rebase_source : 95004a869b40dd72a643c4dbc0b02e96409fb7d8
