Path: utzoo!attcan!uunet!lll-winken!lll-lcc!ames!pasteur!ucbvax!hplabs!oracle!rbradbur
From: rbradbur@oracle.UUCP (Robert Bradbury)
Newsgroups: comp.databases
Subject: Correction of benchmark availability from Oracle.
Keywords: Benchmark availabilty, proprietary information, PACBELL
Message-ID: <274@turbo.oracle.UUCP>
Date: 19 Jul 88 03:37:08 GMT
Organization: ORACLE Corporation, Belmont CA, USA
Lines: 43

I have recently been informed by a manager at PACBELL that Oracle
does not have the results of the benchmarks they did, that the
results of the benchmarks are proprietary and that I should be
shot for suggesting to the people reading this topic that a
public utility could release information favoring one vendor
over another.  All I am told that I can say is that PACBELL
has chosen Oracle as one of its database systems.

Ouch!

Of course that leaves us back where we were before I opened
my mouth and stuck my foot in it.  As I see it the companies
that have the manpower & machine resources to do these
benchmarks "right" are dis-inclined to publish the results
due to the potential legal hassles.  From participating in these
benchmarks I know that large companies spend man-months and days of
dedicated computer time (which is expensive for a machine like
an Amdahl) comparing these systems.  I also know of one large
company with a policy of re-evaluating the RDBMS on the market
every 2 years.  Given that most of the RDBMS are larger than
the UNIX kernel (some by a factor of 2 or more) and probably
require many times more functional tests than UNIX does one
is faced with a massive job when trying to compare these
systems.

I think I'm forced to agree with the suggestion that we
need an open forum where all vendors agree to participate
and publish the results.  I think this was tried at UniForum
recently and that one or more of the major UNIX RDBMS vendors
did not participate (I know we did).  I think problems crop
up here when the UNIX machine vendors willing to donate machine
time happen not to be those machines on which one vendor or
another performs favorably.  The obvious solution is to have
a variety of platforms agreed upon far enough in advance
that the RDBMS vendors have their latest and greatest
running and tuned for those machines.  Also they would have
to agree to publish all of the code used in the benchmarks
so other vendors would be free to respond to any overly
"creative" approaches to superior performance. 

If I recall there was/is a committee/working group for USENIX/UniForum
which is focused on these kinds of things.  Does anyone know what
its status is and/or if the results of the last RDBMS comparison
were published?