Avoid calling libtool to link every single test case, by building just one
binary from all the sources.
This binary is then given the task of choosing tests to run (based on user
selection and individual test requirement), forking each test into its own
process and accumulating the results.
As opposed to 8.61.
I had actually forgotten we had documented this. If I had
remembered I could have put more meaningful messages with
some of my recent updates to ps-specific reference images.
Like many other gradient tests, we're just capturing the buggy
output of poppler here, (though this time I *did* verify with
Adobe acroread that our PDF output seems to be correct).
These capture the current poppler output, (rather than the
ideal output). We're still waiting for poppler to start
using cairo gradients before this test will run through
cleanly. And even with these new reference images, there's
still some translation variance, so the -25 tests still
fail.
Compare the current output against a previous run to determine if there
has been any change since last time, and only run through imagediff if
there has been. For the vector surfaces, we can check the vector output
first and potentially skip the rasterisation. On my machine this reduces
the time for a second run from 6 minutes to 2m30s. As most of the time,
most test output will remain unchanged, so this seems to be a big win. On
unix systems, hard linking is used to reduce the amount of storage space
required - others will see about a three-fold increase in the amount of
disk used. The directory continues to be a stress test for file selectors.
In order to reduce the changes between runs, the current time is no longer
written to the PNG files (justified by that it only exists as a debugging
aid) and the boilerplate tweaks the PS surface so that the creation date
is fixed. To fully realise the benefits here, we need to strip the
creation time from all the reference images...
The biggest problem with using the caches is that different runs of the
test suite can go through different code paths, introducing potential
Heisenbergs. If you suspect that caching is interfering with the test
results, use 'make -C test clean-caches check'.
This is justified by the previously mentioned bug on poppler
gradients. Note that this test only passes with a patch
to fix a bug in poppler (not yet upstream). Here it is:
PATCH: Keep 'cairo_shape' and 'cairo' consistent
https://bugs.freedesktop.org/show_bug.cgi?id=14593
This is justified by the following new bug report:
Poppler should paint images with CAIRO_EXTEND_PAD
https://bugs.freedesktop.org/show_bug.cgi?id=14578
The four affected tests are:
paint-source-alpha
paint-with-alpha
rotate-image-surface-paint
scale-source-surface-paint
Allow using a previous test output directory as a source of
reference images. To make use of this, set the environment
variable 'CAIRO_REF_DIR' to point at an old test directory,
relative to the current test directory.
This is useful for testing backends when reference images haven't
been created yet, or which the current reference image structure
can't accomodate, like multiple font backends.
- Remove cairo_test_expect_failure. cairo-test.c now checks
env var CAIRO_XFAIL_TESTS to see if the running test is
expected to fail. The reason for expected failure is
appended to the test description.
- Test description is written out.
- Failed/crashed tests also write a line out to stderr (in red),
so one can now redirect stdout to /dev/null to only see failures.
- cairo_test() has been changed to not take the draw function
anymore, instead, draw function is now part of the test struct.
- "make check" doesn't allow limiting backends to test using env
var anymore. To limit backends to test, one should use the
TARGETS variable on the make command line.
- "make check-valgrind" now writes its log to valgrind-log instead
of valgrind.log, to not interfere with test log file processing.
Don't AC_SUBST a dozen different FOO_CFLAGS and FOO_LIBS. Instead, incrementally build up just CAIRO_CFLAGS and CAIRO_LIBS.
Don't list flags that should get picked up via dependency information through Requires.
Add description of move_to_show_surface.
