Summary:
Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers.
This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively.
As an example after this change libc++ can implement the `MODULES-DEFINES` simply using:
```
mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST)
parseIntegratedTestScript(test, additional_parsers=[mparser])
macro_list = mparser.getValue()
```
Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs
Subscribers: mgrang, llvm-commits, cfe-commits
Differential Revision: https://reviews.llvm.org/D27005
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@288694 91177308-0d34-0410-b5e6-96231b3b80d8
This shouls now be safe and not break any more bots. It's strictly better to use '--sdk macosx', otherwise xcrun can return weird things for example when you have Command Line Tools or the SDK installed into '/'.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@288385 91177308-0d34-0410-b5e6-96231b3b80d8
Negative exit codes are usually exceptions. They're easier to recognize
in hex. Compare -1073741502 to 0xc0000142.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@286150 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
This will allow us to revert LLD r284768, which added spaces to get MSys
echo to print what we want.
Reviewers: ruiu, inglorion, rafael
Subscribers: modocache, llvm-commits
Differential Revision: https://reviews.llvm.org/D26009
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@285237 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
r283710 introduced two regressions, one to llvm-lit, and the other to
lit executables that were installed via setuptools. Add instructions on
how to test for these regressions in the future.
Reviewers: ddunbar, delcypher, beanz, chapuni, cmatthews, echristo
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25459
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@284919 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
The Python file `utils/lit/lit/ShUtil.py` contains:
1. Logic used by lit itself
2. A set of unit tests for that logic, which can be run by invoking
`python utils/lit/lit/ShUtil.py`
Move these unit tests to a `tests/unit` subdirectory of lit, and run
the tests as part of lit's test suite. This ensures that, should the
lit test suite be included in LLVM's own regression test suite, these
unit tests will also be run.
(Instructions on how to run lit's test suite can be found in
`utils/lit/README.txt`.)
Reviewers: ddunbar, echristo, delcypher, beanz
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D25411
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@283968 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Using Python linter flake8 on the utils/lit reveals several linter
warnings designated "F401: Unused import". Fix or silence these
warnings.
Some of these unused imports are legitimate, while some are part of lit's API.
For example, users of lit expect to be able to access `lit.formats.ShTest` in
their `lit.cfg`, despite the module hierarchy for that symbol actually being
`lit.formats.shtest.ShTest`. To silence linter errors for these lines,
include a "noqa" directive.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D25407
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@283710 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
`TestingProgressDisplay` initializes its `current` attribute to `None`, but
never reads or writes the value again. Remove it.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25415
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@283709 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
`ArgumentError` is not defined by the Python standard library.
Executing this line of code would throw a exception, but not the
intended one. It would throw a `NameError` exception, since `ArgumentError`
is undefined.
Use `ValueError` instead, which is defined by the Python standard
library.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25410
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@283708 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
Semicolons aren't necessary as statement terminators in Python, and
each of these uses are superfluous as they appear at the end of a line.
The convention is to not use semicolons where not needed, so remove them.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25409
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@283707 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
The minimum version of Python required to run LLVM's test suite is 2.7.
Remove a workaround for older Python versions.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25400
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@283705 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
The minimum version of Python necessary to run the LLVM test suite is
2.7. Code to work around Python 2.5 and lower isn't necessary.
Reviewers: ddunbar, echristo, delcypher, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25209
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@283169 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
optparse is deprecated in Python 2.7, which is the minimum version of
Python required to run the LLVM test suite. Replace its usage in lit
with argparse, optparse's 2.7 replacement module.
argparse has several benefits over optparse, but this commit does not
make use of those benefits yet. Instead, it simply uses the new API,
and attempts to keep the number of changes to a minimum.
Confirmed that lit's test suite, as well as LLVM's regression test suite,
still pass with these changes.
Patch By Brian Gesiak!
Reviewers: ddunbar, echristo, beanz, delcypher
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25173
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@283152 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
lit's `OneCommandFileTest` class implements an abstract method that
raises if called. However, it raises by referencing an undefined
symbol. Instead, raise explicitly by throwing a `NotImplementedError`.
This is clearer, and appeases Python linters.
Patch By Brian Gesiak!
Reviewers: ddunbar, echristo, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25170
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@283090 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
In Python, `None` is a singleton, so checking whether a variable is
`None` may be done with `is` or `is not`. This has a slight advantage
over equiality comparisons `== None` and `!= None`, since `__eq__` may
be overridden in Python to produce sometimes unexpected results.
Using `is None` and `is not None` is also recommended practice in
https://www.python.org/dev/peps/pep-0008:
> Comparisons to singletons like `None` should always be done with `is` or
> `is not`, never the equality operators.
Patch by Brian Gesiak!
Reviewers: ddunbar, echristo, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25168
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@283088 91177308-0d34-0410-b5e6-96231b3b80d8
- This is primarily useful as a "fail fast" mode for lit, where it will stop
running tests after the first failure.
- Patch by Max Moiseev.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@282452 91177308-0d34-0410-b5e6-96231b3b80d8
Lots of unittests started failing under asan after r280455. It seems
they've been failing for a long time, but lit silently ignored them.
Downgrade the error so we can figure out what is going on.
Filed http://llvm.org/PR30285.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@280674 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
This is a follow up to r280455, where a check for the process exit code
was introduced. Some ASAN bots throw this error now, but it's impossible
to understand what's wrong with them, and the issue is not reproducible.
Reviewers: vitalybuka
Differential Revision: https://reviews.llvm.org/D24210
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@280550 91177308-0d34-0410-b5e6-96231b3b80d8
Do this by creating a temp directory in the normal system temp
directory, and cleaning it up on exit.
It is still possible for this temp directory to leak if Python exits
abnormally, but this is probably good enough for now.
Fixes PR18335
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@280501 91177308-0d34-0410-b5e6-96231b3b80d8
googletest formatted tests are discovered by running the test executable.
Previously testing would silently succeed if the test executable crashed
during the discovery process. Now testing fails with "error: unable to
discover google-tests ..." if the test executable exits with a non-zero status.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@280455 91177308-0d34-0410-b5e6-96231b3b80d8
Apparently nobody evaluated multiprocessing on Windows since Daniel
enabled multiprocessing on Unix in r193279. It works so far as I can
tell.
Today this is worth about an 8x speedup (631.29s to 73.25s) on my 24
core Windows machine. Hopefully this will improve Windows buildbot cycle
time, where currently it takes more time to run check-all than it does
to self-host with assertions enabled:
http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/20
build stage 2 ninja all ( 28 mins, 22 secs )
ninja check 2 stage 2 ( 37 mins, 38 secs )
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@280382 91177308-0d34-0410-b5e6-96231b3b80d8
- This allows tools like emacs to automatically find the config file path when
you step through errors.
- Patch by Dave Abrahams.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@276357 91177308-0d34-0410-b5e6-96231b3b80d8
This variable is used by ASan (and other sanitizers in the future)
on s390x-linux to override a check for CVE-2016-2143 in the running
kernel (see revision 267747 on compiler-rt). Since the check simply
checks if the kernel version is in a whitelist of known-good versions,
it may miss distribution kernels, or manually-patched kernels - hence
the need for this variable. To enable running the ASan testsuite on
such kernels, this variable should be passed from the environment
down to the testcases.
Differential Revision: http://reviews.llvm.org/D19888
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@273825 91177308-0d34-0410-b5e6-96231b3b80d8
- The intended use of this was just in diagnostics, so we shouldn't pay the
cost of reading these all the time.
- This will avoid including the full output of each command in tests which
fail, but the most important use case for this was to gather the output of
the specific command which failed.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@272365 91177308-0d34-0410-b5e6-96231b3b80d8
Use os.devnull instead of tempfiles when substituting '/dev/null' on
Windows machines. This should make the bots just a bit speedier.
Thanks to Yunzhong Gao for testing this patch on Windows!
Differential Revision: http://reviews.llvm.org/D20549
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@272290 91177308-0d34-0410-b5e6-96231b3b80d8
- This will cause lit to automatically include the first 1K of data in
redirected output files when a command fails (previously if the command
failed, but the main point of the test was, say, a `FileCheck` later on, then
the log wasn't helpful in showing why the command failed).
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@272021 91177308-0d34-0410-b5e6-96231b3b80d8
- This only applies to scripts executed by the _internal_ shell script
interpreter.
- This patch reworks the log to look more like a shell transcript, and be less
verbose (but in the interest of calling attention to the important parts).
Here is an example of the new format, for commands with/without failures and
with/without output:
```
$ true
$ echo hi
hi
$ false
note: command had no output on stdout or stderr
error: command failed with exit status 1
```
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@271610 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
This patch adds a "REQUIRES-ANY" feature test that is disjunctive. This marks a test as `UNSUPPORTED` if none of the specified features are available.
Libc++ has the need to write feature test such as `// REQUIRES-ANY: c++98, c++03` when testing of behavior that is specific to older dialects but has since changed.
Reviewers: rnk, ddunbar
Subscribers: ddunbar, probinson, llvm-commits, cfe-commits
Differential Revision: http://reviews.llvm.org/D20757
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@271468 91177308-0d34-0410-b5e6-96231b3b80d8
Summary:
We need these variables to concatenate two absolute paths to construct
a valid path. Currently, %t\%t is, for example, expanded to C:\foo\C:\foo,
which is not a valid path because ":" is not a valid path character
on Windows. With this patch, %t will be expanded to C\foo.
Differential Revision: http://reviews.llvm.org/D19757
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@268168 91177308-0d34-0410-b5e6-96231b3b80d8
