Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sun-barr!newstop!sun!slovax!lm
From: lm@slovax.Sun.COM (Larry McVoy)
Newsgroups: comp.arch
Subject: Re: IOStone
Message-ID: <144085@sun.Eng.Sun.COM>
Date: 23 Oct 90 22:05:57 GMT
References: <7834@ucdavis.ucdavis.edu> <144018@sun.Eng.Sun.COM> <2783@crdos1.crd.ge.COM>
Sender: news@sun.Eng.Sun.COM
Reply-To: lm@sun.UUCP (Larry McVoy)
Organization: Sun Microsystems, Mountain View
Lines: 76

In article <2783@crdos1.crd.ge.COM> davidsen@crdos1.crd.ge.com (bill davidsen) writes:
>In article <144018@sun.Eng.Sun.COM> lm@sun.UUCP (Larry McVoy) writes:
>| Come on guys, it's obvious that you blew it with this benchmark.  Instead of
>| arguing endlessly about how great it is, why don't you go off and try again?
>
>  The nice thing about this is that you have such a detached attitude
>and have presented a great list of technical flaws ;-) IOstone returns
>results which match *in ratio* measurements of production IO bound
>programs. Therefore many of us find the results useful for comparing
>machine to one another as platforms for running that load.

Your statment that IOstone matches real programs is unsubstantiated.  
I have not been able to find any such program.

You want to understand, apparently, why I don't like this benchmark.
OK, here's why.  IOstone claims to measure IO performance, not cache
performance.  It explicitly tries to clear the cache but fails to do so
in the face of large caches such as those provided by SunOS, Mach, and
Multics.  This gives us results that can not be considered apples to
apples comparisons.

It would be OK if the designers understood the differences between
various caching systems and were trying to measure those differences.
If they were trying to do that they could have done so much more
effectively by

	#!/bin/sh

	SIZE=1024
	while true
	do	BS=1024
		COUNT=`expr $SIZE / $BS`
		dd if=/dev/zero of=/tmp/XXX bs=$BS count=$COUNT > /dev/null
		if [ $? -ne 0 ]
		then	echo can\'t write the file, probably too big.
			exit 0
		fi
		time dd if=/tmp/XXX of=/dev/null > /dev/null
		SIZE=`expr $SIZE * 2`
	done

And printed a graph of machine, filesize, time.  This would clearly show
"who's got a big one" in terms of caches, that is :-)


They showed up at Sun one day and wanted to talk to the file system people
about our "buffer cache" and how it managed to work so fast.  They didn't
realize that anything but buffer cache based file systems existed.  They
*designed in* knowledge of the buffer cache.   They claim their system is
based in file system traces.  I'd like to see the traces.  I'd like to see
the user program that goes out and tries to clear the cache before running.
I've taken lots of file system traces and the traces that I've seen don't
like anything like the IO that results from running IOstone.

If you really trust this benchmark, go buy Suns.  Suns look fantastic
under this benchmark.  Don't blame me when you try out your new Sun and
find that, surprise, it is not 100X faster after all.

I'm fighting this benchmark because it gives you some information and
tells you how that information is to be interpreted.  In particular,
it tells you about data movement and calls that data movement IO rates.

The data that it is moving on some machines may be IO, but on Sun machines
it is almost always data from the cache.  I don't call data in the cache
IO; maybe you do.

If you want an example of a much more reasonable IO benchmark, and dd just
doesn't cut it for you, then take a look at Tim Bray's benchmark, bonnie.
That benchmark measures IO rates.

If you want a benchmark that measures cache hits versus cache misses, you
can do better than IOstone.  You could run MusBus or any number of
light weight multi user benchmarks.  They do a fairly good job of
measuring your caching / paging / swapping system.  
---
Larry McVoy, Sun Microsystems     (415) 336-7627       ...!sun!lm or lm@sun.com