cairo/test

How to use cairo's test suite
=============================
Using this test should be as simple as running:

  meson test -C $builddir -v

assuming that the cairo distribution in the directory above has been
configured and built in $builddir. The test suite here will use the
locally compiled library rather than installed version.

The above command runs some source code as well as the rendering
tests. All the rendering tests are in a single meson test named
"cairo". It can be run with:

  meson test -C $builddir -v cairo

The results of the test suite run are summarized in an index.html file
written to $builddir, which, when viewed in a web browser makes it
quite easy to visually see any failed renderings alongside the
corresponding reference image, (and a diff image as well).

As some browsers do not permit Javascript to read from the local
filesystem, the view-test-results.py script in the $builddir can used
to view the results. It starts a http server serving the current
directory before displaying the test results in a browser.

The test suite needs to be run before any code is committed and before
any release. See below for hints and rules governing the use of the suite.

Running specific tests
----------------------
During development it is desirable to only run a single test or groups
of related tests. Individual tests can be run by specifying the tests
names as arguments to the cairo test.

  meson test -v cairo --test-args "record-neg-extents-unbounded record-neg-extents-bounded"

The test suite is built as a single binary, "cairo-test-suite", which
can be run directly instead of via meson. This may be more convenient
when specifying multiple test arguments. The executable needs to be
run in $builddir/test in order to find the image conversion
executables. Simply running the executable will run all tests. eg

  cd $builddir/test
  ./cairo-test-suite

Normally the executable will use the locally compiled library. But if
you have a LD_LIBRARY_PATH defined or built cairo with an -rpath, the
executable may not link with the locally compiled library.

Individual tests can be run be specifying the tests names as arguments
to cairo-test-suite. For example, to run specific tests:

  ./cairo-test-suite record-neg-extents-unbounded record-neg-extents-bounded

Or if you want to run all paint.* related tests you can use:

  ./cairo-test-suite paint

Or if you want to check the current status of known failures:

  ./cairo-test-suite XFAIL

Or to run a subset of tests, use the -k option to run only the tests
that include the given keyword:

  ./cairo-test-suite -k downscale

The -l option lists all tests available. For convenience, the file
"completion.bash" in $builddir can be sourced to provide bash
completion of the test names in cairo-test-suite.

By default, each test that is run will be tested on all backends that
have been built. The CAIRO_TEST_TARGET environment variable can be
used to select specific backends. For example:

    CAIRO_TEST_TARGET=pdf ./cairo-test-suite text-rotate

Each test is run twice on each backend, once for each format. The
CAIRO_TEST_TARGET_FORMAT environment variable selects a single
format. The formats are "rgba" and "rgb". For example to run the
text-rotate test only on the image backend and only for the "rgba"
format:

  CAIRO_TEST_TARGET=image CAIRO_TEST_TARGET_FORMAT=rgba ./cairo-test-suite text-rotate

Running tests under modified environments or tools
--------------------------------------------------
To run tests under in gdb, one can simply run the cairo-suite-suite in
gdb. Usually a specific test will be desired. eg

  CAIRO_TEST_TARGET=image CAIRO_TEST_TARGET_FORMAT=rgba gdb --args ./cairo-test-suite text-rotate

The cairo test suite runs each test in a child process so that crashes
do not stop the test. When running the test suite under tools, the "-f"
option disables forking. eg to run a test under valgrind:

  CAIRO_TEST_TARGET=image CAIRO_TEST_TARGET_FORMAT=rgba valgrind ./cairo-test-suite -f text-rotate

The cairo test suite detects if it is running under gdb and disables
forking. Hence the reason the gdb example above did not use "-f".

Complete list of test options
-----------------------------
cairo-test-suite options:

  ./cairo-test-suite [-afkxsl] [test-names|keywords ...]

  -a    all; run the full set of tests. By default the test suite
          skips similar surface and device offset testing.
  -f    foreground; do not fork
  -k    match tests by keyword
  -l    list only; just list selected test case names without executing
  -s    include slow, long running tests
  -x    exit on first failure

Environment Variables:

CAIRO_TEST_TARGET
  Run the test only the the listed targets. eg
    CAIRO_TEST_TARGET="image"
    CAIRO_TEST_TARGET="pdf,ps3,ps2"

CAIRO_TEST_TARGET_FORMAT
  Run the test only the the listed targets. eg
    CAIRO_TEST_TARGET_FORMAT="rgba"
    CAIRO_TEST_TARGET_FORMAT="rgba,rgb"

CAIRO_TEST_MODE
  May contain a list of one or more of the following options
    "full"            - equivalent to the "-a" option
    "similar"
    "offset"
    "scale"
    "foreground"      - equivalent to the "-f" option
    "exit-on-failure" - equivalent to the "-x" option
  eg
    CAIRO_TEST_MODE="full,foreground"

CAIRO_TESTS
  A list of test to run. This is equivalent to listing the tests as
  arguments to cairo=test-suite. eg
    CAIRO_TESTS="record-neg-extents-unbounded record-neg-extents-bounded"
  The tests may be separated by any of " \t,:;".

srcdir
  The test suite needs to find the "test" directory in the source
  tree. The srcdir environment variable can be used to specify the
  location of this directory. If this environment variable is not set,
  the binary looks for the directory "srcdir" in the current
  directory. The meson build system symlinks "srcdir" in the
  $builddir/test directory to the "test" directory in the source
  tree. If this is not found the binary defaults to the current
  directory.

CAIRO_REF_DIR
  The directory containing the reference images. By default the test
  suite looks for "reference" in $srcdir. For convenience, you can set
  CAIRO_REF_DIR to point at a previous test directory, relative to the
  current test directory, and any previous output will be used by
  preference as reference images.

CAIRO_TEST_TIMEOUT
  The maximum time, in seconds, an indivdual test may run before it is
  stopped. The default is 60 seconds.

Getting the elusive zero failures
---------------------------------
It's generally been very difficult to achieve a test run with zero
failures. The difficulties stem from the various versions of the many
libraries that the test suite depends on, (it depends on a lot more
than cairo itself), as well as fonts and other system-specific
settings. If your system differs significantly from the system on
which the reference images were generated, then you will likely see
the test suite reporting "failures", (even if cairo is working just
fine).

Since the test suite must pass the Continuous Integration test in the
Gitlab instance hosting the cairo repository, the easiest way to get a
new or updated test to pass is to extract the test output from the
cairo gitlab CI to use as reference images.

What if I can't make my system match?
-------------------------------------
For one reason or another, you may be unable to get a clean run of the
test suite even if cairo is working properly, (for example, you might
be on a system without freetype). In this case, it's still useful to
be able to determine if code changes you make to cairo result in any
regressions to the test suite. But it's hard to notice regressions if
there are many failures both before and after your changes.

For this scenario, you can capture the output of a run of the test
suite before your changes, and then use the CAIRO_REF_DIR environment
variable to use that output as the reference images for a run after
your changes. The process looks like this:

  # Before code change there may be failures we don't care about
  cd $builddir/test
  ./cairo-test-suite

  # Let's save those output images
  mkdir /some/directory/
  cp -r output /some/directory/

  # hack, hack, hack

  # Now to see if nothing changed:
  CAIRO_REF_DIR=/some/directory/ ./cairo-test-suite

Best practices for cairo developers
===================================
If we all follow the guidelines below, then both the test suite and
cairo itself will stay much healthier, and we'll all have a lot more
fun hacking on cairo.

Before creating a Merge Request
-------------------------------
After pushing your branch to your cairo fork on gitlab.freedesktop.org,
check that the CI passes. A MR cannot be merged until the CI passes.

It is likely that you will need to download test output images from
the Gitlab CI to use as reference images in order to get CI to pass.

The reference image filenames have the form:

<test name>.*.ref.png for correct images that pass.

<test name>.*.xfail.png for incorrect images due to bugs.

The <test name>.ref.png is the most general image that all backends
and formats for the <test name> test will be required to match.

When comparing test output to the reference images, the test suite
performs a perceptual diff which permits some minor pixel differences
that are not visually discernible. Some targets may have differences
that exceed the threshold of the perceptual diff but still appear the
same as the reference image when eyeballed. In these cases, target
specific reference images can be created that override the general
reference image for the specific target.

Reference images for specific backends can be created using filenames of the form:
  <test name>.<target>.ref.png

eg clip.pdf.ref.png

The target name "ps" can be used for both "ps2" and "ps3" targets.

Reference images for specific formats can be created using filenames of the form:
  <test name>.<target>.<format>.ref.png

eg
  clip.pdf.argb32.ref.png
  clip.pdf.rgb24.ref.png

The recommended practice for creating reference images for a new test is:

1. Create a <test name>.ref.png reference image from <test name>.image.argb32.out.png
output.

2. Push to gitlab and check the CI result.

3. If the image target fails, copy the <test name>.image.argb32.out.png output
from CI and use as the reference image.

4. Once the image target passes in CI, check that the other targets
pass. If any fail, create <test name>.<target>.ref.png reference images if the
output appears correct.

5. If a target output is incorrect, try to determine if the cairo
target output is correct. eg view pdf files in Adobe Reader. If the
cairo target output is correct, it can be assumed the image converson
from target to png is buggy. File a bug against the conversion tool
and create a <test name>.<target>.xfail.png reference image.

In all cases, before creating a reference image from test output, the
image should be viewed to ensure that it is correct.

When a new bug is found
-----------------------
A new test case should be added by imitating the style of an existing
test. This means adding the following files:

	new-bug.c
	reference/new-bug.ref.png
	reference/new-bug.xfail.png

Where new-bug.c is a minimal program to demonstrate the bug, following
the style of existing tests. The new-bug.ref.png image should contain
the desired result of new-bug.c if the bug were fixed while
new-bug.xfail.png contains the current results of the test.

"meson.build" should be edited by adding new-bug.c to test_sources.
And last but not least, don't forget to "git add" the new files.

When a new feature is added
---------------------------
It's important for the regression suite to keep pace with development
of the library. So a new test should be added for each new feature.
The work involved is similar the work described above for new bugs.
The only distinction is that the test is expected to pass so it
should not need a new-bug.xfail.png file.

While working on a test
-----------------------
Before a bugfix or feature is ready, it may be useful to compare
output from different builds. For convenience, you can set
CAIRO_REF_DIR to point at a previous test directory, relative
to the current test directory, and any previous output will be
used by preference as reference images.

When a bug is fixed
-------------------
The fix should be verified by running the test suite which should
result in an "unexpected pass" for the test of interest. Rejoice as
appropriate, then remove the relevant xfail.png file from git.

Before releasing
----------------
All tests should return a result of PASS for all supported (those enabled by
default) backends, meaning all known bugs are fixed, resulting in the happy
message:

	All XX tests passed

Some notes on limitations in poppler
====================================
One of the difficulties of our current test infrastructure is that we
rely on external tools to convert cairo's vector output (PDF,
PostScript, and SVG), into an image that can be used for the image
comparison. This means that any bugs in that conversion tool will
result in false negatives in the test suite.

Here's the rule: If a cairo-pdf test reports a failure, but viewing
the resulting PDF file with acroread suggests that the PDF itself is
correct, then there's likely a bug in poppler. In this case, we can
simply report the poppler bug. Once we've done this, we can capture
poppler's buggy output as a pdf-specific reference image (as
reference/*.xfail.png) so that the test suite will regard the test as
passing, (and we'll ensure there is no regression).

Once the poppler bug gets fixed, the test suite will start reporting a
false negative again, and this will be easy to fix by simply removing
the pdf-specific reference image.