This commit covers one of the left out modifications from
"commit cd11a4ff0421fd293279b202be800550890574bb" by Bryce.
It removes the duplicate macro definition in cairo-perf-diff-files.c
which by default includes the cairo-perf.h having MAX macro defnition.
Signed-off-by: Ravi Nanjundappa <nravi.n@samsung.com>
Having spent the last dev cycle looking at how we could specialize the
compositors for various backends, we once again look for the
commonalities in order to reduce the duplication. In part this is
motivated by the idea that spans is a good interface for both the
existent GL backend and pixman, and so they deserve a dedicated
compositor. xcb/xlib target an identical rendering system and so they
should be using the same compositor, and it should be possible to run
that same compositor locally against pixman to generate reference tests.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
P.S. This brings massive upheaval (read breakage) I've tried delaying in
order to fix as many things as possible but now this one patch does far,
far, far too much. Apologies in advance for breaking your favourite
backend, but trust me in that the end result will be much better. :)
The cairo-perf-diff-files tool would ignore perf reports with
just one test for no apparent reason. The traces take so long
that it's useful to be able to compare runs with just one trace.
As the change and ranking is based on the min_ticks, and as this can
sometimes deviate wildly from median_ticks, include min_ticks in the
output.
In particular it helps to explain cases like:
xlib-rgba rectangles_similar-rgba-mag_source-512 10.13 88.41% -> 5.77 0.19%: 1.50x slowdown
which becomes
xlib-rgba rectangles_similar-rgba-mag_source-512 3.83 (10.13 88.41%) -> 5.75 (5.77 0.19%): 1.50x slowdown
(Considering the poor standard deviation on the initial measurement, this
is more likely a sampling error than a true regression.)
More the large slowdowns to the end. This has two pleasing effects:
1. There is symmetry between large speedups at the top, and large
slowdowns at the bottom, with long bars -> short bars -> long bars.
2. After a cairo-perf-diff run the largest slowdowns are immediately
visible on the console. What better way to flag performance
regressions?
Generate a cairo-perf-diff graph for a series of commits in order to be
able to identify significant commits. Still very crude, but minimally
functional.
The compiler complained about passing a non-string literal as the format
to printf, so just to sanitize the code and keep the compiler happy, add
the magic "%s" format.
Previously, if the change in the first test case was small enough
to be considered insignificant, then the header lines showing the
names of the old and new configurations would be omitted. This
commit fixes that bug.
Now, if you pass exactly two performance reports on the command line
you'll get the traditional report style again, (so the tool remains
backwards compatible). If you really want the new style with two
reports you can get it by adding /dev/null as a third argument.
This support is intended to compare the identical backends across multiple
reports from several different configurations, (of one sort or another).
The configuration names used in the report are taken from the filenames
of the report files, (which will format most nicely if 8 characters or
less).
The traditional two-input report mode, (showing one line perdiff with
all speedups before all slowdowns), is removed with this commit, but
is intended to return again shortly.
We terminate the iteration by adding a final report with a NULL
name. This will simplify future code that iterates over more than
two reports simultaneously.
An upcoming change will allow cairo-perf-diff-files to generate
a report based on more than two files. This prepares for that by
moving the minimum-change support from the 3rd positional argument
to a new --min-change option.
Ideally, the minimum value would indicate the best possible performance,
but I've encountered situations where a bogus minimum value gets lodged
into the cached results for some reason, (and yet doesn't also get
discarded as an outlier). The downside of that situation is that running
more iterations never changes the result, so it's hard to fix the problem,
(resulting in cairo-perf-diff feeling just plain broken as more runs
change nothing).
So let's try using the median time instead.
This patch adds options to disable the UTF-8 change bars and replace
them with ASCII '*' gfx. The motivation is that UTF-8 really does some
damage to some terminals, and some always forget to pipe stuff through
a pager to make it safe. The new options for cairo-perf-diff-files are:
--no-utf
--no-bars
Instead of just discarding the worst 15% of the results, we now
do IQR-based detection and elimination of outliers. Also, instead
of reporting mean times we now report minimum and median times.
We still do compute the mean and standard deviation for the
detection of when results seem stable for early bailout. And we
do still report the standard deviation.
A statistician might complain that it looks funny to report the
median and the standard deviation together, (instead of median
and average absolute deviation from the median, say), but I liked
the standard deviation since it is always larger, so it might
ensure better separatation if we use it to determine when two
sets of results are sufficiently different to be interesting.
The old cairo-perf-diff is now named cairo-perf-diff-files, but
the new one calls out to it and can still be used in an identical
way.
The new cairo-perf-diff can also be used to see what the performance
impact of a single commit is like so:
cairo-perf-diff HEAD
or between two commits:
cairo-perf-diff 1.2.4 HEAD
The script is careful to always run the latest cairo-perf program
even when testing old versions of the library. Also, the output
from any given performance run is cached so it gets less painful
to run as the cache gets primed (the cache is in .perf next to
.git).
The script is still a bit fragile in spots. In particular it depends
on cairo-perf being built in advance but doesn't do anythin to ensure
that happens.
