cairo/test/README

Regression test suite for cairo.

Using this test should be as simple as running:

	make check

assuming that the cairo distribution in the directory above has been
configured and built. The test suite here goes through some effort to
run against the locally compiled library rather than any installed
version, but those efforts may fall short depending on the level of your
libtool madness.

The test suite needs to be run before any code is committed and before
any release. Here are the rules governing the use of the suite:


Tailoring tests running
-----------------------

There are some mechanisms to limit the tests run during "make check".
These come very handy when doing development, but should not be used
to circumvent the "pass" requirements listed below.

To limit the backends that the tests are run against, use the
TARGETS make variable, that can also be passed to make.
It should contain a (space-, comma-, etc-separated) list of backends to test.
To limit the tests run, use the TESTS make variable, which should be a
space-separated list of tests to run.  For example:

  make check TARGETS=image,ps TESTS="zero-alpha"


Before committing
-----------------

All tests should return a result of PASS or XFAIL. The XFAIL results
indicate known bugs. The final message should be one of the following:

	All XX tests behaved as expected (YY expected failures)
	All XX tests passed

If any tests have a status of FAIL, then the new code has caused a
regression error which should be fixed before the code is committed.


When a new bug is found
-----------------------
A new test case should be added by imitating the style of an existing
test. This means adding the following files:

	new-bug.c
	new-bug-ref.png

Where new-bug.c is a minimal program to demonstrate the bug, following
the style of existing tests. The new-bug-ref.png image should contain
the desired result of new-bug.c if the bug were fixed.

Makefile.am should be edited, adding new-bug.c to both the TESTS and
XFAIL_TESTS lists and new-bug-ref.png to EXTRA_DIST. Add new-bug to
.gitignore, and last but not least, don't forget to "git add" the new
files.


When a new feature is added
---------------------------
It's important for the regression suite to keep pace with development
of the library. So a new test should be added for each new feature.
The work involved is similar the work described above for new bugs.
The only distinction is that the test is expected to pass so it
should not be added to the XFAIL_TESTS list.

While working on a test
-----------------------
Before a bugfix or feature is ready, it may be useful to compare
output from different builds. For convenience, you can set
CAIRO_REF_DIR to point at a previous test directory, relative
to the current test directory, and any previous output will be
used by preference as reference images.


When a bug is fixed
-------------------
The fix should be verified by running the test suite which should
result in an "unexpected pass" for the test of interest. Rejoice as
appropriate, then remove the relevant file name from the XFAIL_TESTS
variable in Makefile.am.


Before releasing
----------------
All tests should return a result of PASS for all supported (those enabled by
default) backends,  meaning all known bugs are fixed, resulting in the happy
message:

	All XX tests passed

Some notes on limitations in poppler
------------------------------------
One of the difficulties of our current test infrastructure is that we
rely on external tools to convert cairo's vector output (PDF,
PostScript, and SVG), into an image that can be used for the image
comparison. This means that any bugs in that conversion tool will
result in false negatives in the test suite.

We've identified several such bugs in the poppler library which is
used to convert PDF to an image. This is particularly discouraging
because 1) poppler is free software that will be used by *many* cairo
users, and 2) poppler calls into cairo for its rendering so it should
be able to do a 100% faithful conversion.

So we have an interest in ensuring that these poppler bugs get fixed
sooner rather than later. As such, we're trying to be good citizens by
reporting all such poppler bugs that we identify to the poppler
bugzilla. Here's a tracking bug explaining the situation:

	Poppler does not yet handle everything in the cairo test suite
	https://bugs.freedesktop.org/show_bug.cgi?id=12143

Here's the rule: If a cairo-pdf test reports a failure, but viewing
the resulting PDF file with acroread suggests that the PDF itself is
correct, then there's likely a bug in poppler. In this case, we can
simply report the poppler bug, (making it block 12143 above), post the
PDF result from the test suite, and list the bug in this file. Once
we've done this, we can capture poppler's buggy output as a
pdf-specific reference image so that the test suite will regard the
test as passing, (and we'll ensure there is no regression).

Once the poppler bug gets fixed, the test suite will start reporting a
false negative again, and this will be easy to fix by simply removing
the pdf-specific reference image.

Here are the reported poppler bugs and the tests they affect:

Poppler doesn't correctly handle gradients with transparency
https://bugs.freedesktop.org/show_bug.cgi?id=12144
--------------------------------------------------
gradient-alpha
linear-gradient
trap-clip
linear-gradient
linear-gradient-reflect

Poppler renders patterned text as black
https://bugs.freedesktop.org/show_bug.cgi?id=14577
--------------------------------------------------
text-pattern

Poppler should paint images with CAIRO_EXTEND_PAD
https://bugs.freedesktop.org/show_bug.cgi?id=14578
--------------------------------------------------
paint-source-alpha
paint-with-alpha
rotate-image-surface-paint
scale-source-surface-paint