Instead of just discarding the worst 15% of the results, we now
do IQR-based detection and elimination of outliers. Also, instead
of reporting mean times we now report minimum and median times.
We still do compute the mean and standard deviation for the
detection of when results seem stable for early bailout. And we
do still report the standard deviation.
A statistician might complain that it looks funny to report the
median and the standard deviation together, (instead of median
and average absolute deviation from the median, say), but I liked
the standard deviation since it is always larger, so it might
ensure better separatation if we use it to determine when two
sets of results are sufficiently different to be interesting.
This test is really just for hammering the double to fixed-point conversion
(in _cairo_fixed_from_double) that happens as doubles from API calls gets
translated into internal cairo fixed-point numbers.
Many thanks to Josh Triplett for help with theses fixes:
1. Simplify the usage of git to eliminate the fragile and nasty
stuff we were doing previously, (such as manually symlinking
things under .git rather than just using git-clone -s).
2. Don't try running latest cairo-perf with LD_LIBRARY_PATH
pointing to built cairo version. Apparently stupid libtool
and its use of rpath is foiling us here. Instead just run
whatever cairo-perf gets built as part of the source that
gets checked out.
This second point means that cairo-perf-diff won't yet be useful
for comparing the performance of old cairo revisions that pre-date
cairo-perf. I've since been reasing and we might be able to use
--disable-rpath to the configure script to get what we want.
The old cairo-perf-diff is now named cairo-perf-diff-files, but
the new one calls out to it and can still be used in an identical
way.
The new cairo-perf-diff can also be used to see what the performance
impact of a single commit is like so:
cairo-perf-diff HEAD
or between two commits:
cairo-perf-diff 1.2.4 HEAD
The script is careful to always run the latest cairo-perf program
even when testing old versions of the library. Also, the output
from any given performance run is cached so it gets less painful
to run as the cache gets primed (the cache is in .perf next to
.git).
The script is still a bit fragile in spots. In particular it depends
on cairo-perf being built in advance but doesn't do anythin to ensure
that happens.
This makes the entire performance test suite about 10 times faster
on my system. And I don't think that the results are significantly
worse, (many tests are stable after only 5 iterations while some
still run to 100 iterations without reaching our stability criteria).
This will finally allow us to very easily add lots of other
tests that will similarly involve iterating over the various
sources and operators of interest.
This commit begins to show the benefits of the reorganization
from the last few commits. Specifically, we don't need a callback
for every run anymore, and now that the cairo_t* is available to
the paint entry function and the size changes only on the outer
loop, (rather than inside cairo_perf_run), there's no need for
the ugly, static cache for the source surface.
The motivation here is to have the cairo_t context available
to the perf funcs before they call into cairo_perf_run, (so
that they can do one-time setup of source etc. for several
runs).
This effectively reverts the change made in:
3c407aa80a
The justification is that we want to be able to see the
trends of increasing the image size while keeping the test
the same. But changing the number of iterations in a size-
specific way interferes with that. Also, the standard deviation
is almost always better this way, (though it now varies more
by size).
And as a bonus, the total time required to run the suite is now
less.
This changes the perf test output format to be a little more human friendly,
reporting times in ms instead of seconds. It also adds a test number
that could be used in the future for specifying an explicit test to run
(test number, target surface, test name, and size uniquiely identify
a test).
Also adds a few paint tests.
We do this by adding a new cairo_perf_timer_set_finalize function and
in the case of the xlib backend passing a callback to that function
that does a 1x1 XGetImage.
Add a new cairo_boilerplate_mode_t so that the boilerplate targets can
do slightly different things if being tested for correctness vs. being
run for performance.
I've seen this improve the std. deviation often by a factor of 2
and occasionally up to a factor of 10. It is sometimes not much
better, but never seems to be appreciably worse compared to using
gettimeofday.
Thanks to David A. Schleef <ds@schleef.org> and his liboil for
the implementation.
