Ideally, the minimum value would indicate the best possible performance,
but I've encountered situations where a bogus minimum value gets lodged
into the cached results for some reason, (and yet doesn't also get
discarded as an outlier). The downside of that situation is that running
more iterations never changes the result, so it's hard to fix the problem,
(resulting in cairo-perf-diff feeling just plain broken as more runs
change nothing).
So let's try using the median time instead.
This patch adds options to disable the UTF-8 change bars and replace
them with ASCII '*' gfx. The motivation is that UTF-8 really does some
damage to some terminals, and some always forget to pipe stuff through
a pager to make it safe. The new options for cairo-perf-diff-files are:
--no-utf
--no-bars
Instead of just discarding the worst 15% of the results, we now
do IQR-based detection and elimination of outliers. Also, instead
of reporting mean times we now report minimum and median times.
We still do compute the mean and standard deviation for the
detection of when results seem stable for early bailout. And we
do still report the standard deviation.
A statistician might complain that it looks funny to report the
median and the standard deviation together, (instead of median
and average absolute deviation from the median, say), but I liked
the standard deviation since it is always larger, so it might
ensure better separatation if we use it to determine when two
sets of results are sufficiently different to be interesting.
The old cairo-perf-diff is now named cairo-perf-diff-files, but
the new one calls out to it and can still be used in an identical
way.
The new cairo-perf-diff can also be used to see what the performance
impact of a single commit is like so:
cairo-perf-diff HEAD
or between two commits:
cairo-perf-diff 1.2.4 HEAD
The script is careful to always run the latest cairo-perf program
even when testing old versions of the library. Also, the output
from any given performance run is cached so it gets less painful
to run as the cache gets primed (the cache is in .perf next to
.git).
The script is still a bit fragile in spots. In particular it depends
on cairo-perf being built in advance but doesn't do anythin to ensure
that happens.
