arkcompiler_runtime_core/tests/cts-generator

Test generator and test runner tools

Tests are being generated during building.

• CTS_TEST_SELECT_OPTION - options, passed to test-runner.rb. Useful for defining --exclude-tag and --include-tag options.
• PANDA_CTS_JOBS_NUMBER - amount of parallel jobs for test execution. Default is 8.

Test generator

Options:

Usage: generate-cts.rb [options]
    -t, --template FILE              Path to template yaml file to generate tests (required)
    -s, --schema FILE                Path to json schema for template yaml (required)
    -k, --skip                       Skip yaml schema validation
    -o, --output DIR                 Path to directory where tests will be generated (required)
    -h, --help                       Prints this help

Usage example:

 ${PANDA_SRC_ROOT}/tests/cts-generator/generate-cts.rb \
   -t ${PANDA_SRC_ROOT}/tests/cts-generator/cts-template/template.yaml \
   -s ${PANDA_SRC_ROOT}/tests/cts-generator/cts-template/yaml-schema.json \
   -o cts-generated

This command will generate CTS tests using provided template file. Template is validated using schema file.

Test runner

Options:

Usage: test-runner.rb [options]
    -p, --panda-build DIR            Path to panda build directory (required)
    -t, --test-dir DIR               Path to test directory to search tests recursively, or path to single test (required)
    -v, --verbose LEVEL              Set verbose level 1..5
        --verbose-verifier           Allow verifier to produce extended checking log
        --aot-mode                   Perform AOT compilation on test sources
        --timeout SECONDS            Set process timeout, default is 30 seconds
        --dump-timeout SECONDS       Set process completion timeout, default is 30 seconds
        --enable-core-dump           Enable core dumps
        --verify-tests               Run verifier against positive tests (option for test checking)
        --with-quickener             Run quickener tool after assembly
        --global-timeout SECONDS     Set testing timeout, default is 0 (ulimited)
    -a, --run-all                    Run all tests, ignore "runner-option: ignore" tag in test definition
        --run-ignored                Run ignored tests, which have "runner-option: ignore" tag in test definition
        --reporter TYPE              Reporter for test results (default 'log', available: 'log', 'jtr', 'allure')
        --report-dir DIR             Where to put results, applicable for 'jtr' and 'allure' logger
        --verifier-config PATH       Path to verifier config file
    -e, --exclude-tag TAG            Exclude tags for tests
    -o, --panda-options OPTION       Panda options
    -i, --include-tag TAG            Include tags for tests
    -b, --bug_id BUGID               Include tests with specified bug ids
    -j, --jobs N                     Amount of concurrent jobs for test execution (default 8)
        --prlimit OPTS               Run panda via prlimit with options
        --plugins PLUGINS            Paths to runner plugins
    -h, --help                       Prints this help

Usage example:

${PANDA_SRC_ROOT}/tests/cts-generator/test-runner.rb
   -t cts-generated \
   -p ${PANDA_BUILD_ROOT} \
   -e release -e debug \
   -i sanitizer-fail,wrong-tag

This command will start all tests in cts-generated directory. Tests which have runner options ignore will be ignored. Tests that have release and sanitizer-fail will be excluded.

To run all tests, add -a options.

To run only tests with ignore runner option, add --run-ignored options.

Tips

How to specify options for cmake cts-generator target?

CTS_TEST_SELECT_OPTION variable can be used.

cmake -DCTS_TEST_SELECT_OPTION="-b 1316 -a" ../panda

Run all tests marked with bugid 1316.

How to generate all test not using cmake/make

cd ${ROOT_PATH}/tests/cts-generator
./generate-cts.rb \
   -t ./cts-template/template.yaml \
   -s ./cts-template/yaml-schema.json \
   -o cts-generated

How to run all tests

All test can be executed using make cts-generated command, test with ignore runner options will be ignored by test runner. If you want to run all tests, you can do the following:

cmake ${ROOT_PATH} -DCTS_TEST_SELECT_OPTION="--run-all"
make cts-generator

Also you can start test-runner.rb directly:

cd ${BUILD_DIR}/tests/cts-generator
./test-runner.rb
  -p ${BUILD_DIR} \
  -t ./cts-generated/ \
  -v 2 -j 8 \
  -i release \
  -e sanitizer-fail

Tests with release tag will be included to test execution, with sanitizer-fail will be excluded.

How to run test with specified bug id runner-option?

Example:

test-runner.rb \
  -t ./cts-generated/ \
  -p ${BUILD_DIR} \
  -b 977 \
  -v 2 \
  -a

Please note that -a options (--run-all) is defined, otherwise tests will be excluded, if they have ignore runner option.

How run panda via prlimit?

Example:

test-runner.rb \
  -t ./cts-generated/ \
  -p ${BUILD_DIR} \
  --prlimit='--stack=8000000:8000000 --nproc=512'

What should I do with failed/passed tests after bug-fixing?

1. Run all tests using regular build or using tests or cts-geerator targets.

2. Run tests related to bug.

   cmake -DCTS_TEST_SELECT_OPTION="-b 1316 -a" ../panda
   make cts-generator
   

3. If all test passed, congrats! Now you can enable tests for regular execution. Update all tests that have bugid: [number] and ignore: true - remove bugid relation and ignore tag to allow test runner to execute test.

4. If some tests failed, you can update them, if there are problems with tests. If tests are correct, continue with bugfixing and repeat all steps.