mirror of
https://gitlab.freedesktop.org/cairo/cairo.git
synced 2026-01-02 19:20:15 +01:00
77e9b05b2b
Allow using a previous test output directory as a source of reference images. To make use of this, set the environment variable 'CAIRO_REF_DIR' to point at an old test directory, relative to the current test directory. This is useful for testing backends when reference images haven't been created yet, or which the current reference image structure can't accomodate, like multiple font backends.
96 lines
3.3 KiB
Text
96 lines
3.3 KiB
Text
Regression test suite for cairo.
|
|
|
|
Using this test should be as simple as running:
|
|
|
|
make check
|
|
|
|
assuming that the cairo distribution in the directory above has been
|
|
configured and built. The test suite here goes through some effort to
|
|
run against the locally compiled library rather than any installed
|
|
version, but those efforts may fall short depending on the level of your
|
|
libtool madness.
|
|
|
|
The test suite needs to be run before any code is committed and before
|
|
any release. Here are the rules governing the use of the suite:
|
|
|
|
|
|
Tailoring tests running
|
|
-----------------------
|
|
|
|
There are some mechanisms to limit the tests run during "make check".
|
|
These come very handy when doing development, but should not be used
|
|
to circumvent the "pass" requirements listed below.
|
|
|
|
To limit the backends that the tests are run against, use the
|
|
TARGETS make variable, that can also be passed to make.
|
|
It should contain a (space-, comma-, etc-separated) list of backends to test.
|
|
To limit the tests run, use the TESTS make variable, which should be a
|
|
space-separated list of tests to run. For example:
|
|
|
|
make check TARGETS=image,ps TESTS="zero-alpha"
|
|
|
|
|
|
Before committing
|
|
-----------------
|
|
|
|
All tests should return a result of PASS or XFAIL. The XFAIL results
|
|
indicate known bugs. The final message should be one of the following:
|
|
|
|
All XX tests behaved as expected (YY expected failures)
|
|
All XX tests passed
|
|
|
|
If any tests have a status of FAIL, then the new code has caused a
|
|
regression error which should be fixed before the code is committed.
|
|
|
|
|
|
When a new bug is found
|
|
-----------------------
|
|
A new test case should be added by imitating the style of an existing
|
|
test. This means adding the following files:
|
|
|
|
new-bug.c
|
|
new-bug-ref.png
|
|
|
|
Where new-bug.c is a minimal program to demonstrate the bug, following
|
|
the style of existing tests. The new-bug-ref.png image should contain
|
|
the desired result of new-bug.c if the bug were fixed.
|
|
|
|
Makefile.am should be edited, adding new-bug.c to both the TESTS and
|
|
XFAIL_TESTS lists and new-bug-ref.png to EXTRA_DIST. Add new-bug to
|
|
.gitignore, and last but not least, don't forget to "git add" the new
|
|
files.
|
|
|
|
|
|
When a new feature is added
|
|
---------------------------
|
|
It's important for the regression suite to keep pace with development
|
|
of the library. So a new test should be added for each new feature.
|
|
The work involved is similar the work described above for new bugs.
|
|
The only distinction is that the test is expected to pass so it
|
|
should not be added to the XFAIL_TESTS list.
|
|
|
|
While working on a test
|
|
-----------------------
|
|
Before a bugfix or feature is ready, it may be useful to compare
|
|
output from different builds. For convenience, you can set
|
|
CAIRO_REF_DIR to point at a previous test directory, relative
|
|
to the current test directory, and any previous output will be
|
|
used by preference as reference images.
|
|
|
|
|
|
When a bug is fixed
|
|
-------------------
|
|
The fix should be verified by running the test suite which should
|
|
result in an "unexpected pass" for the test of interest. Rejoice as
|
|
appropriate, then remove the relevant file name from the XFAIL_TESTS
|
|
variable in Makefile.am.
|
|
|
|
|
|
Before releasing
|
|
----------------
|
|
All tests should return a result of PASS for all supported (those enabled by
|
|
default) backends, meaning all known bugs are fixed, resulting in the happy
|
|
message:
|
|
|
|
All XX tests passed
|
|
|