How To Use Mesa's test tool: deqp-runner

deqp-runner is a series of tools written by mesa developers for running vulkan and opengl quality test case programs. It can run in parallel and robustly dEQP (draw-element quality program), piglit, SkQP (Skia Quality Program), and so on.

In my experience, when running large dEQP test cases, your own changes may cause umd (user mode driver) to fail, crash, timeout, or hang. If you simply use deqp-vk, the khronos testcases program for vulkan, the test will stop when umd throws unrecoverable error. And we cannot easily get the results that new failures based on the previous version.

Yes, we don’t need to write a new / own scripts to handle testcase stopping. deqp-runner help us to do it:

• Parallel running cases, it’s very useful if we need to run a huge set
• Run next case and report this case’s state if an unrecoverable error is thrown
• automatically compare the differences between baseline and current version

Build

The tool is written in rust-lang, and is very easy to get and build (compare with C++, LoL).

git clone https://gitlab.freedesktop.org/mesa/deqp-runner.git
cd deqp-runner
cargo build --release --target-dir _build

The executable file is generated in _build/release, deqp-runner and piglit-runner are very useful for me.

Usage for dEQP

We run dEQP cases via deqp-runner:

deqp-runner run \
    [runner options...] \
    -- \
    [deqp program options...]

For running dEQP testcases, we have some important arguments in deqp-runner:

deqp

dEQP executable file

jobs

how many threads running in parallel

output

where the results are stored

timeout

the run is terminted if the given number of seconds is exceeded

caselist

a list of testcase list files

env

a list of test run-time environment variables, e.g. VK_DRIVER_FILES, MESA_LOADER_DRIVER_OVERRIDE

If you run a base driver first, you can set baseline to failures of base, deqp-runner will automatically compare current results with baseline.

If you just want to run a subset of given case lists, include-tests help us to do it, testcases are skipped if testcase name is not match in this option.

For example, I use command a lot:

deqp-runner run \
    --deqp deqp-vk \
    --caselist {binding-mode,descriptor-indexing}.txt compute.txt image/*.txt \
    --env VK_DRIVER_FILES=vulkan/icd.d/radeon_icd.x86_64.json \
    --jobs $(awk -v CPUS=$(lscpu -e |wc -l) 'BEGIN{print int((CPUS-1)*0.8)}') \
    --output deqp-vk_$(date --iso-8601="date") \
    --timeout 240.0 \
    -- \
    --deqp-log-images=disable \
    --deqp-log-shader-sources=disable \
    --deqp-log-decompiled-spirv=disable \
    --deqp-shadercache=disable

Running opengl dEQP is similar to vulkan:

deqp-runner run \
    --deqp glcts \
    --caselist gl/khronos_mustpass/*-main.txt \
    --env LD_LIBRARY_PATH=opengl MESA_LOADER_DRIVER_OVERRIDE=radeonsi \
    --jobs 4 \
    --output deqp-gl_$(date --iso-8601="date") \
    --timeout 240.0 \
    -- \
    --deqp-gl-config-name=rgba8888d24s8ms0 \
    --deqp-surface-{height,width}=256 \
    --deqp-visibility=hidden

Output

The main output files we should focus on:

• results.csv that all testcases status in here
• failures.csv that all failures, crashes, timeouts in here

According to source code, the test status as following:

If using baseline, maybe get new status:

If any results came back with an unexpected failure, run the caselist again to see if we get the same results, and mark any changing results as flaky tests. In results file, it called Flake.

Easily report the flake testcases in current running:

awk -F, '$2 == "Flake"{print $1}' results.csv >flakes.txt

Usage for piglit

Similar to deqp-runner, piglit must set profile mode what is *.xml.gz in tests folder. And using piglit-folder instead of deqp executable file.

For example:

piglit-runner run \
    --jobs 2 --profile quick --timeout 300 \
    --piglit-folder piglit \
    --output piglit_$(date --iso-8601="date") \
    --

Different with other runner, testcase’s name looks strange in result file. For example, asmparsertest@arbfp1.0@cos-03.txt. In fact, the name is from profile.