Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!magnus.acs.ohio-state.edu!tut.cis.ohio-state.edu!sei.cmu.edu!fs7.ece.cmu.edu!o.gp.cs.cmu.edu!netnews
From: af@cs.cmu.edu (Alessandro Forin)
Newsgroups: comp.os.mach
Subject: Re: The value of microbenchmarking (WAS Re: Mach RPC Throughput...)
Message-ID: <1991Mar25.160017.28358@cs.cmu.edu>
Date: 25 Mar 91 16:00:17 GMT
Sender: netnews@cs.cmu.edu (USENET News Group Software)
Organization: School of Computer Science, Carnegie Mellon
Lines: 38

> Isn't a comparison of UNIX emulations just as much a microbenchmark as
> null RPC times?  

I have not mentioned Unix in my post at all, and it was not assumed in any
whatsoever way.  As a matter of fact, we can pass most of those tests with
our MS-DOS emulator as well :-)) But you'll agree with me that an operating
system that cannot fulfill the needs of its users (by whatever means and
standards) is not an operating system but something else.  To qualify as a
"general purpose multiuser operating system" I believe the tests I mentioned
can be considered the bare minimum, esp in these days of heavy networking.

That having been said, the answer to your question is no.  What I have
described is a functionality test, not a performance test.  After a system
passes the functionality tests [e.g. we are sure it is a complete system,
there won't be surprises in it for users] we can go off to performance
testing as much as you wish and have a lot of fun in the process too.
Usually, microbenchmarks tend to be significant only for knowledgeable
people [e.g. those who have implemented a message passing kernel understand
the implications of certain numbers better than others], while
macrobenchmarks such as a large compilation or something are easily
understood and useful to many more people.  But each one is good in its own
way.

To those who have done it, comparing two full-blown Unix emulations is by no
mean a microbenchmark.
As for the value of it.. it is my experience that there is a lot to be
learned and understood by doing that also.  We have tried to convey this and
other impressions in the USENIX paper, in the same conference there was a
paper of the V kernel group on the same subject: by reading both papers I
think one can get a better but still partial feeling of the work involved
and of its value, merits and limits.

BTW, I did not mean the list of OSes in the previous post to be complete in
any whatsoever way, just a couple of examples of other "research operating
systems" with which meaningful comparisons viz Mach have been made.
There are others, and they know who they are.

sandro-