Although cairo-perf is not written to perform explicit failure testing of
cairo, it does generate long sequences of cairo operations which often
trigger unexpected errors. By including it with make check, it becomes
even easier for the programmer to check that one has not broken cairo in
terms of expected behaviour or performance.
Ideally, the minimum value would indicate the best possible performance,
but I've encountered situations where a bogus minimum value gets lodged
into the cached results for some reason, (and yet doesn't also get
discarded as an outlier). The downside of that situation is that running
more iterations never changes the result, so it's hard to fix the problem,
(resulting in cairo-perf-diff feeling just plain broken as more runs
change nothing).
So let's try using the median time instead.
The text perf-case tries to fill the region with a single text string,
but fails to detect when the current point does not advance due to an
error. This causes the perf-case to enter an infinite loop, so we break
out when the cairo_status() has been set.
cairo-perf and the X server should be bound to CPUs (either the same
or separate) on SMP systems. Not doing so causes random results when
the X server is moved to or from cairo-perf's CPU during the
benchmarks.
When choosing the samples that are not outliers we use a half-open interval
(outlier_min <= x < outlier_max). This causes all of the samples to be
discarded when the interquartile range is 0 because none of them are less
than outlier_max. Fix the problem and make the test more consistent by
using a closed interval (outliner_min <= x <= outlier_max).
This patch adds options to disable the UTF-8 change bars and replace
them with ASCII '*' gfx. The motivation is that UTF-8 really does some
damage to some terminals, and some always forget to pipe stuff through
a pager to make it safe. The new options for cairo-perf-diff-files are:
--no-utf
--no-bars
This test shows that drawing a 100x100 single-pixel wide box outline is
currently 5 to 16 times slower when using the natural cairo_stroke() as
compared to a rather awkward cairo_fill() of two rectangles.
[ # ] backend-content test-size min(ticks) min(ms) median(ms) stddev. iterations
[ 0] image-rgba box-outline-stroke-100 301321 0.218 0.219 0.39% 5
[ 1] image-rgba box-outline-fill-100 18178 0.013 0.013 0.43% 5
[ 0] xlib-rgba box-outline-stroke-100 379177 0.275 0.276 1.39% 6
[ 1] xlib-rgba box-outline-fill-100 83355 0.060 0.060 0.17% 5
Before this change, cairo-perf-diff would work correctly only
if run from the top-level directory, (and if run from any other
directory it would not pull in the latest boilerplate and perf
files).
Previously, we were using rsync to recursively copy all files
from boilerplate and perf when doing the performance tests. This
had the fatal flaw that pre-built binaries would be hard coded to
link against the cairo library from $CAIRO_DIR with an absolute
path, (due to libtool rpath stuff).
By only copying git-managed files, we only get source and the compiled
binaries (or libtool wrappers) will now be hard-coded to link against
the library under test.
With the fancy new, incremental cairo-perf-diff we don't want to keep
resetting the same working tree back and forth between the old and new
versions and rebuilding everything all over again. So use two different
build directories instead of one.
This fixes the rebuild of the library itself, but the perf stuff is still
being rebuilt, (since it's being re-copied each time).
cairo-perf-diff now accepts a -f command-line option to force it to
re-generate performance results even if something exists in the cache
already. It also now uses raw mode and appends to the cached file
rather than rewriting the results.
Finally, it also now allows a -- option on the command line and passes
all the subsequent command-line options to cairo-perf. This is handy for
limiting cairo-perf to run only on a subset of the tests of interest.
The map for this test case was originally demonstrated as a
performance problem in this mozilla bug report:
A very slow SVG file with <path>s
https://bugzilla.mozilla.org/show_bug.cgi?id=332413
I obtained permission from the creator of the original file to
include the data here, (see comments in world-map.h for details).
We don't need this at this deep level since callers can now
implement this limiting manually since stats.iterations is
now returned. Also, this was interfering with the -i option
to cairo-perf anyway.
