Avoid calling libtool to link every single test case, by building just one
binary from all the sources.
This binary is then given the task of choosing tests to run (based on user
selection and individual test requirement), forking each test into its own
process and accumulating the results.
In order to run under memfault, the framework is first extended to handle
running concurrent tests - i.e. multi-threading. (Not that this is a
requirement for memfault, instead it shares a common goal of storing
per-test data). To that end all the global data is moved into a per-test
context and the targets are adjusted to avoid overlap on shared, global
resources (such as output files and frame buffers). In order to preserve
the simplicity of the standard draw routines, the context is not passed
explicitly as a parameter to the routines, but is instead attached to the
cairo_t via the user_data.
For the masochist, to enable the tests to be run across multiple threads
simply set the environment variable CAIRO_TEST_NUM_THREADS to the desired
number.
In the long run, we can hope the need for memfault (runtime testing of
error paths) will be mitigated by static analysis. A promising candidate
for this task would appear to be http://hal.cs.berkeley.edu/cil/.
The following four tests are disabled:
gradient-alpha, linear-gradient, text-pattern, trap-clip
We don't use XFAIL as that would disable all backends, (but
we can still usefully use these tests on backends other than
PDF).
- Remove cairo_test_expect_failure. cairo-test.c now checks
env var CAIRO_XFAIL_TESTS to see if the running test is
expected to fail. The reason for expected failure is
appended to the test description.
- Test description is written out.
- Failed/crashed tests also write a line out to stderr (in red),
so one can now redirect stdout to /dev/null to only see failures.
- cairo_test() has been changed to not take the draw function
anymore, instead, draw function is now part of the test struct.
- "make check" doesn't allow limiting backends to test using env
var anymore. To limit backends to test, one should use the
TARGETS variable on the make command line.
- "make check-valgrind" now writes its log to valgrind-log instead
of valgrind.log, to not interfere with test log file processing.
Shrink linear-gradient and text-pattern test outuput.
The real motivation for this is to hide an annoying difference in the
gradient output that occurs when under the influence of a device
offsets. My opinion is that the gradient output should not change due
to just an integer offset, but we currently don't have a precise
gradient specification, so this minor deviation really isn't relevant.
A better fix for this might have been to make the test suite more
forgiving of tiny differences, (perhaps on a per-test basis), but
another advantage of shrinking these tests is that they now run a lot
faster.
It is true that these tests now also exercise fewer different output
values, but that's not the designed purpose of the tests so I don't
feel bad about losing that characteristic.
