Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!rutgers!ub.d.umn.edu!umnd-cs-gw!cthombor
From: cthombor@umnd-cs-gw.d.umn.edu (Clark Thomborson)
Newsgroups: comp.compression
Subject: Re: 1) decompression speeds, 2) rounding.
Keywords: data compression rounding speeds
Message-ID: <1378@ub.d.umn.edu>
Date: 27 Jun 91 15:36:14 GMT
References: <894@spam.ua.oz>
Sender: news@ub.d.umn.edu
Reply-To: cthombor@umnd-cs-gw.UUCP (Clark Thomborson)
Organization: University of Minnesota, Duluth Computer Science
Lines: 63
Nntp-Posting-Host: gw.d.umn.edu

As Ross guessed, the question of how to ``average'' a series of performance
figures has indeed been studied a lot.

My reply is based on Chapters 1 and 2 of Hennessy & Patterson's new text,
{\it Computer Architecture: A Quantitative Approach}, Morgan-Kaufmann, 1990.
This book has lots of great stuff in it, beyond its discussion of
performance averaging, including a chapter on what I expect is the next
``standard'' workstation architecture, a RISC cpu with a Cray-like
floating-point pipe.  No, I am not getting a kickback on their sales!

In brief, the hard part of performance averaging is to figure out
the application.  It makes sense to report the total time required to
process a representative benchmark dataset.  Perhaps this would be a
directory with a representative mix of files.  This view lays bare the
inherent problem in interpreting ``average performance'': if you measure
your program's runtime on a directory that contains a lot of C source files,
and my intended use is for decompressing a single binary boot file, your
performance figure is of little use to me.

I disagree with Hennessy & Patterson on relatively minor points,
in that I think one should report mins and maxes, as well as averages.
I also like speed figures (Kbytes/second in the data compression business,
MIPS in the architecture business) more than they seem to: I can interpret
speeds more or less directly into a *rough* runtime estimate for my own
application, but I can't always figure out the appropriate scaling factor
to convert their ``total time'' into a runtime estimate for my application.
For example, if you reported the range of decompression speeds you observed
on various types of files, I would probably be able to decide if your
compressor is fast enough for my binary boot file application.  If, instead,
you reported just a total time (or an average speed) for compressing a
directory with a ``representative mix'' of binaries, text files, and
picture files, I would be worried.  Perhaps the binaries ran a lot faster
or slower than the others (for some reason).  In this case, the total time
or average speed would not tell me much about my application.  If all
the files run at about the same speed, reporting mins and maxes will
reveal this; totals and averages will leave me in the dark.

Hennessy and Patterson point out that ``total time'' has a significant
advantage: it's relatively hard to cheat on its definition.  For example,
the MIPS rating of a cpu has become discredited, due to the widespread
practice of choosing ones' benchmark to maximize the reported figure.
In your context, reporting total time would dispose of the question of
whether one should measure decompression speed as time/input_size or
time/output_size.  I would argue that you should still report speeds,
but *clearly state your definition*.  You should also report your runtime
conditions: cpu, clock speed, computer manufacturer and model number,
operating system name and version number, compiler name and version number,
compiler optimization setting, etc.  If you are measuring compression
performance on a directory full of files, the IO configuration of your
system becomes important: at a minimum, you should disclose whether
you are using a floppy disk or a hard disk, but the disk model number
and its bus type (SCSI? ESDI?) are also important.

As for your question on ``deep'' versus ``shallow'' averages, the accepted
practice is to carry out all calculations at high precision, then round
just before printing.  As you noticed, this means your published data will
not always look consistent, although it does mean that every number is as
accurately reported as possible.  The only exception I have noticed to
this rule is that an accountant will sometimes fudge the rounding of a
number, so that the printed (and accurately rounded) total is consistent
with the sum of the (perhaps inaccurately rounded) terms.

							Clark