Similar to cairo-test, we free any global memory used by cairo for its
caches through the debug interfaces. We do this so that valgrind does
not unnecessary warn about memory leaks for the cached data and any true
leak is then not lost in the noise.
Previously, if the change in the first test case was small enough
to be considered insignificant, then the header lines showing the
names of the old and new configurations would be omitted. This
commit fixes that bug.
Now, if you pass exactly two performance reports on the command line
you'll get the traditional report style again, (so the tool remains
backwards compatible). If you really want the new style with two
reports you can get it by adding /dev/null as a third argument.
This support is intended to compare the identical backends across multiple
reports from several different configurations, (of one sort or another).
The configuration names used in the report are taken from the filenames
of the report files, (which will format most nicely if 8 characters or
less).
The traditional two-input report mode, (showing one line perdiff with
all speedups before all slowdowns), is removed with this commit, but
is intended to return again shortly.
We terminate the iteration by adding a final report with a NULL
name. This will simplify future code that iterates over more than
two reports simultaneously.
An upcoming change will allow cairo-perf-diff-files to generate
a report based on more than two files. This prepares for that by
moving the minimum-change support from the 3rd positional argument
to a new --min-change option.
Previously only "--" would givethe usage, and not "--help"
as intended. The new approach has been tested with ash, bash,
and dash, (hopefully that's good enough).
Although cairo-perf is not written to perform explicit failure testing of
cairo, it does generate long sequences of cairo operations which often
trigger unexpected errors. By including it with make check, it becomes
even easier for the programmer to check that one has not broken cairo in
terms of expected behaviour or performance.
Ideally, the minimum value would indicate the best possible performance,
but I've encountered situations where a bogus minimum value gets lodged
into the cached results for some reason, (and yet doesn't also get
discarded as an outlier). The downside of that situation is that running
more iterations never changes the result, so it's hard to fix the problem,
(resulting in cairo-perf-diff feeling just plain broken as more runs
change nothing).
So let's try using the median time instead.
The text perf-case tries to fill the region with a single text string,
but fails to detect when the current point does not advance due to an
error. This causes the perf-case to enter an infinite loop, so we break
out when the cairo_status() has been set.
cairo-perf and the X server should be bound to CPUs (either the same
or separate) on SMP systems. Not doing so causes random results when
the X server is moved to or from cairo-perf's CPU during the
benchmarks.
When choosing the samples that are not outliers we use a half-open interval
(outlier_min <= x < outlier_max). This causes all of the samples to be
discarded when the interquartile range is 0 because none of them are less
than outlier_max. Fix the problem and make the test more consistent by
using a closed interval (outliner_min <= x <= outlier_max).
