As we use cairo to convert SVG files back to an image, that process is
dependent upon changes within our library and so we cannot skip the
conversion if the SVG file happens to match a previous run. Fortunately,
librsvg is quick enough that this is not a major issue.
For this we extend the boilerplate get_image() routines to extract a
single page out of a paginated document and then proceed to manually
check each page of the fallback-resolution test.
(Well that's the theory, in practice SVG doesn't support multiple pages
and so we just generate a new surface for each resolution. But the
infrastructure is in place so that we can automate other tests,
e.g. test/multi-pages.)
One possibility for a read failure whilst converting the image is if the
external utility crashed. This information is important for the test suite
as knowing input that causes the converter to crash is just as vital as
identifying a crash within the library.
I renamed those generated files to Makefile.*.features but forgot to
update ignore lists. Carl already added the new ones, but didn't
remove the old ones.
I know that I didn't create these Makefile.win32.features files,
so I assume that they are the result of Behdad's build magic and
that he just forgot to add them to .gitignore.
This reverts commit a341cb5a98.
The change introduced in that commit should not be needed and libtool
should just do the right thing. I cannot reproduce the problem
Chris was having no matter how hard I tried.
With --enable-gcov, make check aborts with gcov errors on check-link - it
appears that -lgcov is magic and requires explicit invovation on the
command-line.
Quick summary of changes:
- Move list of cairo source files out of src/Makefile.am and into
src/Sources.mk,
- Generate files src/Config.mk and src/Config.mk.win32 that choose
the right set of source files and headers based on configured
backends and features. This drastically simplifies building
using other build systems. The src/Makefile.win32 file needs
to be updated to reflect these changes.
- Add README files to various directories,
- Add toplevel HACKING file.
Compare the current output against a previous run to determine if there
has been any change since last time, and only run through imagediff if
there has been. For the vector surfaces, we can check the vector output
first and potentially skip the rasterisation. On my machine this reduces
the time for a second run from 6 minutes to 2m30s. As most of the time,
most test output will remain unchanged, so this seems to be a big win. On
unix systems, hard linking is used to reduce the amount of storage space
required - others will see about a three-fold increase in the amount of
disk used. The directory continues to be a stress test for file selectors.
In order to reduce the changes between runs, the current time is no longer
written to the PNG files (justified by that it only exists as a debugging
aid) and the boilerplate tweaks the PS surface so that the creation date
is fixed. To fully realise the benefits here, we need to strip the
creation time from all the reference images...
The biggest problem with using the caches is that different runs of the
test suite can go through different code paths, introducing potential
Heisenbergs. If you suspect that caching is interfering with the test
results, use 'make -C test clean-caches check'.
In order to achieve substantial speed improvements the external conversion
utilities are rewritten as a daemon that communicates with the test suite
over a local socket. This is faster as it avoids the libtool and dynamic
linker overhead for each invocation, the caches persist between tests and
we no longer require a round trip through libpng.
The daemon is started automatically by the test suite and if communication
cannot be established then it falls back to using a pipe to a normal
conversion utility. The daemon will then persist for 60 seconds waiting
for further connections.
Of course any memory leak (stares at poppler) is exacerbated.
If the external conversion utility was killed by a signal (e.g. the user
sent SIGINT), raise that signal within our process as well. This means
that a crash inside poppler or rsvg will be flagged as a crash inside the
test suite, and makes interrupting the test suite far more responsive.
As Behdad suggested, we can dramatically speed up the test suite by
short-circuiting the write to a png file, only to then immediately read it
back in. So for the raster based surfaces, we avoid the round-trip through
libpng by implementing a new boilerplate method to directly extract the image
buffer from the test result. A secondary speedup is achieved by caching the
most recent reference image.
Construct the test name to pass to the boilerplate creation routines such
that it uniquely identifies the test in terms of test, target, content and
pass (similar, offset, thread). This allows the vector targets to create
output different output files for each test, whereas before, later tests
would overwrite existing files making debugging more difficult.
In order to run under memfault, the framework is first extended to handle
running concurrent tests - i.e. multi-threading. (Not that this is a
requirement for memfault, instead it shares a common goal of storing
per-test data). To that end all the global data is moved into a per-test
context and the targets are adjusted to avoid overlap on shared, global
resources (such as output files and frame buffers). In order to preserve
the simplicity of the standard draw routines, the context is not passed
explicitly as a parameter to the routines, but is instead attached to the
cairo_t via the user_data.
For the masochist, to enable the tests to be run across multiple threads
simply set the environment variable CAIRO_TEST_NUM_THREADS to the desired
number.
In the long run, we can hope the need for memfault (runtime testing of
error paths) will be mitigated by static analysis. A promising candidate
for this task would appear to be http://hal.cs.berkeley.edu/cil/.
Convert the boilerplate specific flattened content value to the ordinary
CAIRO_CONTENT_COLOR_ALPHA for use with cairo_push_group_with_content() -
otherwise cairo rightfully flags an error and the test harness decides
that the similar surface is not available.
We had several pdf tests disabled waiting for this bug fix:
Poppler does not correctly handle knockout groups
https://bugs.freedesktop.org/show_bug.cgi?id=12185
That's in place for poppler now, so we're turning the tests
back on. Some of the affected tests now pass perfectly:
over-above-source
over-around-source
over-below-source
over-between-source
Some just needed new reference images:
operator-clear
clip-operator-pdf-argb32
The remaining tests still fail, but none of the failures can
obviously be ascribed to just poppler problems:
clip-operator-pdf-rgb24
operator-source
unbounded-operator
The first two have some serious problems, while in the case
of unbounded-operator the problem is extremely minor (a white
grid appears in the background where the reference image is
all black).
