From 9f58275f98f8b89b5bb9fc179ed7383cd8a78c7d Mon Sep 17 00:00:00 2001 From: Guilherme Gallo Date: Wed, 5 Jan 2022 13:35:18 -0300 Subject: [PATCH] ci: skqp: Add documentation on how to maintain skqp jobs Signed-off-by: Guilherme Gallo Reviewed-by: Emma Anholt Part-of: --- docs/ci/index.rst | 12 ++++++ docs/ci/skqp.rst | 101 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 113 insertions(+) create mode 100644 docs/ci/skqp.rst diff --git a/docs/ci/index.rst b/docs/ci/index.rst index 1cb90f79b17..7f494c7538d 100644 --- a/docs/ci/index.rst +++ b/docs/ci/index.rst @@ -242,3 +242,15 @@ directory. You can hack on mesa and iterate testing the build with: .. code-block:: console sudo docker run --rm -v `pwd`:/mesa $IMAGE ninja -C /mesa/_build + + +Conformance Tests +----------------- + +Some conformance tests require a special treatment to be maintained on Gitlab CI. +This section lists their documentation pages. + +.. toctree:: + :maxdepth: 1 + + skqp diff --git a/docs/ci/skqp.rst b/docs/ci/skqp.rst new file mode 100644 index 00000000000..607bc983e4d --- /dev/null +++ b/docs/ci/skqp.rst @@ -0,0 +1,101 @@ +skqp +==== + +`skqp `_ stands for SKIA Quality +Program conformance tests. Basically, it has sets of rendering tests and unit +tests to ensure that `SKIA `_ is meeting its design specifications on a specific +device. + +The rendering tests have support for GL, GLES and Vulkan backends and test some +rendering scenarios. +And the unit tests check the GPU behavior without rendering images. + +Tests +----- + +Render tests design +^^^^^^^^^^^^^^^^^^^ + +It is worth noting that `rendertests.txt` can bring some detail about each test +expectation, so each test can have a max pixel error count, to tell skqp that it +is OK to have at most that number of errors for that test. See also: +https://github.com/google/skia/blob/main/tools/skqp/README_ALGORITHM.md + +.. _test-location: + +Location +^^^^^^^^ + +Each `rendertests.txt` and `unittest.txt` file must be located inside a specific +subdirectory inside skqp assets directory. + ++--------------+--------------------------------------------+ +| Test type | Location | ++==============+============================================+ +| Render tests | `${SKQP_ASSETS_DIR}/skqp/rendertests.txt` | ++--------------+--------------------------------------------+ +| Unit tests | `${SKQP_ASSETS_DIR}/skqp/unittests.txt` | ++--------------+--------------------------------------------+ + +The `skqp-runner.sh` script will make the necessary modifications to separate +`rendertests.txt` for each backend-driver combination. As long as the test files are located in the expected place: + ++--------------+----------------------------------------------------------------------------------------------+ +| Test type | Location | ++==============+==============================================================================================+ +| Render tests | `${MESA_REPOSITORY_DIR}/src/${GPU_DRIVER}/ci/${GPU_VERSION}-${SKQP_BACKEND}_rendertests.txt` | ++--------------+----------------------------------------------------------------------------------------------+ +| Unit tests | `${MESA_REPOSITORY_DIR}/src/${GPU_DRIVER}/ci/${GPU_VERSION}_unittests.txt` | ++--------------+----------------------------------------------------------------------------------------------+ + +Where `SKQP_BACKEND` can be: + +- gl: for GL backend +- gles: for GLES backend +- vk: for Vulkan backend + +Example file +"""""""""""" + +.. code-block:: console + + src/freedreno/ci/freedreno-a630-skqp-gl_rendertests.txt + +- GPU_DRIVER: `freedreno` +- GPU_VERSION: `freedreno-a630` +- SKQP_BACKEND: `gl` + +.. _rendertests-design: + +skqp reports +------------ + +skqp generates reports after finishing its execution, they are located at the job +artifacts results directory and are divided in subdirectories by rendering tests +backends and unit +tests. The job log has links to every generated report in order to facilitate +the skqp debugging. + +Maintaining skqp on Mesa CI +--------------------------- + +skqp is built alongside with another binary, namely `list_gpu_unit_tests`, it is +located in the same folder where `skqp` binary is. + +This binary will generate the expected `unittests.txt` for the target GPU, so +ideally it should be executed on every skqp update and when a new device +receives skqp CI jobs. + +1. Generate target unit tests for the current GPU with :code:`./list_gpu_unit_tests > unittests.txt` + +2. Run skqp job + +3. If there is a failing or crashing unit test, remove it from the corresponding `unittests.txt` + +4. If there is a crashing render test, remove it from the corresponding `rendertests.txt` + +5. If there is a failing render test, visually inspect the result from the HTML report + - If the render result is OK, update the max error count for that test + - Otherwise, or put `-1` in the same threshold, as seen in :ref:`rendertests-design` + +6. Remember to put the new tests files to the locations cited in :ref:`test-location`