From 6f479e597c3c9e7e4d8a2e1340f6285fb54b27f3 Mon Sep 17 00:00:00 2001 From: Daniel Dunbar Date: Tue, 10 Nov 2009 02:41:27 +0000 Subject: [PATCH] lit: Start documentation testing architecture. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@86655 91177308-0d34-0410-b5e6-96231b3b80d8 --- docs/CommandGuide/lit.pod | 116 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) diff --git a/docs/CommandGuide/lit.pod b/docs/CommandGuide/lit.pod index 97c3e578e6b..1650742cea6 100644 --- a/docs/CommandGuide/lit.pod +++ b/docs/CommandGuide/lit.pod @@ -36,6 +36,9 @@ Finally, B also supports additional options for only running a subset of the options specified on the command line, see L<"SELECTION OPTIONS"> for more information. +Users interested in the B architecture or designing a B testing +implementation should see L<"LIT ARCHITECTURE"> + =head1 GENERAL OPTIONS =over @@ -222,6 +225,119 @@ Depending on the test format tests may produce additional information about their status (generally only for failures). See the L section for more information. +=head1 LIT INFRASTRUCTURE + +This section describes the B testing architecture for users interested in +creating a new B testing implementation, or extending an existing one. + +B proper is primarily an infrastructure for discovering and running +arbitrary tests, and to expose a single convenient interface to these +tests. B itself doesn't contain know how to run tests, rather this logic is +defined by I. + +=head2 TEST SUITES + +As described in L<"TEST DISCOVERY">, tests are always located inside a I. Test suites serve to define the format of the tests they contain, the +logic for finding those tests, and any additional information to run the tests. + +B identifies test suites as directories containing I or +I files (see also B<--config-prefix>. Test suites are initially +discovered by recursively searching up the directory hierarchy for all the input +files passed on the command line. You can use B<--show-suites> to display the +discovered test suites at startup. + +Once a test suite is discovered, its config file is loaded. Config files +themselves are just Python modules which will be executed. When the config file +is executed, two important global variables are predefined: + +=over + +=item B + +The global B configuration object (a I instance), which defines +the builtin test formats, global configuration parameters, and other helper +routines for implementing test configurations. + +=item B + +This is the config object (a I instance) for the test suite, +which the config file is expected to populate. The following variables are also +available on the I object, some of which must be set by the config and +others are optional or predefined: + +B I<[required]> The name of the test suite, for use in reports and +diagnostics. + +B I<[required]> The test format object which will be used to +discover and run tests in the test suite. Generally this will be a builtin test +format available from the I module. + +B The filesystem path to the test suite root. For out-of-dir +builds this is the directory that will be scanned for tests. + +B For out-of-dir builds, the path to the test suite root inside +the object directory. This is where tests will be run and temporary output files +places. + +B A dictionary representing the environment to use when executing +tests in the suite. + +B For B test formats which scan directories for tests, this +variable as a list of suffixes to identify test files. Used by: I, +I. + +B For B test formats which substitute variables into a test +script, the list of substitutions to perform. Used by: I, I. + +B Mark an unsupported directory, all tests within it will be +reported as unsupported. Used by: I, I. + +B The parent configuration, this is the config object for the directory +containing the test suite, or None. + +B The config is actually cloned for every subdirectory inside a test +suite, to allow local configuration on a per-directory basis. The I +variable can be set to a Python function which will be called whenever a +configuration is cloned (for a subdirectory). The function should takes three +arguments: (1) the parent configuration, (2) the new configuration (which the +I function will generally modify), and (3) the test path to the new +directory being scanned. + +=back + +=head2 TEST DISCOVERY + +Once test suites are located, B recursively traverses the source directory +(following I) looking for tests. When B enters a +sub-directory, it first checks to see if a nest test suite is defined in that +directory. If so, it loads that test suite recursively, otherwise it +instantiates a local test config for the directory (see L<"LOCAL CONFIGURATION +FILES">). + +Tests are identified by the test suite they are contained within, and the +relative path inside that suite. Note that the relative path may not refer to an +actual file on disk; some test formats (such as I) define "virtual +tests" which have a path that contains both the path to the actual test file and +a subpath to identify the virtual test. + +=head2 LOCAL CONFIGURATION FILES + +When B loads a subdirectory in a test suite, it instantiates a local test +configuration by cloning the configuration for the parent direction -- the root +of this configuration chain will always be a test suite. Once the test +configuration is cloned B checks for a I file in the +subdirectory. If present, this file will be loaded and can be used to specialize +the configuration for each individual directory. This facility can be used to +define subdirectories of optional tests, or to change other configuration +parameters -- for example, to change the test format, or the suffixes which +identify test files. + +=head2 LIT EXAMPLE TESTS + +The B distribution contains several example implementations of test suites +in the I directory. + =head1 SEE ALSO L