* Move the versions.mk infrastructure to scripts/submodules so it can be used elsewhere too.
* Add a conditional submodule for llvm.
* Fix commit messages emitted by the bump- targets.
* Use a anonymous git url for llvm.
We were seeing timeouts in the runtime step if two builds happen to be scheduled
at the exact same time on the ARM workers on Jenkins, causing increased load and
longer test times. They aborted the runtime step right before it'd finish.
Bump the timeout a bit to avoid this.
On the new Jenkins infrastructure with the new ARM64 bots we're hitting the 2h runtime test
limit right before they'd be finished. Bump the timeout a bit to fix this.
babysitter generates some temp paths in python when running under cygwin, which breaks down
when those paths are passed to the native Windows mono.exe during testing:
Exception in TestRunnerThread ---> System.IO.DirectoryNotFoundException: Could not find a part of the path "C:\cygdrive\c\WINDOWS\TEMP\tmpC4K5yK\babysitter_report_current_test_file.txt".
Some Jenkins workers (e.g. the s390x one) have an earlier python version installed where the babysitter doesn't work.
Fall back to timeout in those cases.
dmcs was deprecated in favor of mcs a while back, yet people are still creating new scripts with it.
Adding a message that shows up on each dmcs invocation should hopefully make this more obvious.
scripts/babysitter stdout logging improvements
Always print stdout log statements, even when all babysitter features
are turned off, even when the test suite does not support the protocol.
The babysitter script was failing to act as a GNU timeout replacement
because it was delivering signals to the spawned process only, whereas
GNU timeout would create a process group. This lead to a problem where
timeout kills by the babysitter would leave children of the spawned
process still running. To fix this, the babysitter script now follows
the exact steps that GNU timeout does (or as close as is possible in
Python). An attempt at a Windows process-group implementation is also
included, but I have not tested it.
Also changed the 'retry_support' key to 'babysitter_protocol' in the
JSON logs, because it was pointed out the old key was misleading (it
is true even when retries are deactivated).
- Using “iteration” for # of retries instead of ambiguous “passes”
- More clearly document what environment keys are for in-script
- Remove CRASH_RESUME key, make setting always true
- Print explicit human-readable messages for stdout log
I see two problems with our current automated PR tests:
- Many false negatives— PR tests often fail due to recurring failures
unrelated to the PR
- Information about what failed is awkward to retrieve, scattered in
multiple places
This commit adds a Python script, `scripts/babysitter`. Currently our
Jenkins test suites are each run wrapped with the `timeout` command.
The babysitter acts as a drop-in replacement for GNU timeout but adds
the following features:
- Logs machine readable output about each test suite (as
line-delimited json)
- If the test suite uses NUnit, can detect if a test case failed or
crashed (terminated mono in mid-test), and retry unsuccessful tests
(up to a limit).
The reasoning here is that tests which fail inconsistently are
currently most likely due to one of our outstanding recurring failures
rather than the change made in the PR. Therefore, if a failing test
succeeds on retry, the PR itself is probably valid (although the
failure should be logged and looked at).
In addition to the script itself, changes to NUnit were required in
order to support the retry feature and allow fine-grained logging for
NUnit suites.
Major TODOs:
- Add retry support for our non-NUnit-based test suites
- Save the XML files NUnit produces (since reruns stomp XMLs from
previous runs)
- Add some kind of sensible feature for dealing with timeouts
The intention is to make build failures easy to spot instead of silently upgrading
target framework from 2 to 4 and then fail down the road on e.g type ambiguity
caused by new 4.0 framework types or methods