Path: utzoo!attcan!uunet!lll-winken!lll-lcc!pyramid!eric
From: eric@pyrps5 (Eric Bergan)
Newsgroups: comp.databases
Subject: Re: Correction of benchmark availability from Oracle.
Keywords: Benchmark availabilty, proprietary information, PACBELL
Message-ID: <31823@pyramid.pyramid.com>
Date: 19 Jul 88 16:15:01 GMT
Sender: daemon@pyramid.pyramid.com
Reply-To: eric@pyrps5.UUCP (Eric Bergan)
Organization: Pyramid Technology Corp., Mountain View, CA
Lines: 50

In article <274@turbo.oracle.UUCP> rbradbur@oracle.UUCP (Robert Bradbury) writes:
>
>I think I'm forced to agree with the suggestion that we
>need an open forum where all vendors agree to participate
>and publish the results.  I think this was tried at UniForum
>recently and that one or more of the major UNIX RDBMS vendors
>did not participate (I know we did).

	Actually, it was held at UNIX Expo in October of 1987. The
participants were Focus, MDBS, Oracle, Progress, and Unify. Other
vendors, based on previous experiences with this event, declined
to attend. The event was run by Neal Nelson, who markets a benchmark
suite that is not database related.

	The test database was only 14 megabytes of data. The test
machine had 16 megabytes of memory, so large portions of the tests
did no disk accesses. The tests were single thread, and somewhat
similar to the DeWitt tests in what they were testing (various
size scans, joins, aggregates, etc.) There were only a few update
tests, and it was not specified if logging was to be enabled. There were
no OLTP-type tests.

	Of the 16 tests, the vendors completed 10, 11, 8, 7, and
14 of the tests (not in any order of vendor). The reason was insufficient
time to allow each vendor to complete.

	All in all, I'm not surprised that the database vendors are
not particularly interested in running in this environment. My understanding
is that Neal Nelson is negotiating with the various DBMS vendors to
provide a better test environment for the upcoming UNIX Expo. But
it will be difficult in a trade show environment, and in the course
of a few days, to be able to do a reasonable test of even a subset of
the 20+ DBMS's available under UNIX. Ideally, you would want to
use a 100+ Mbyte database, which probably kills a day per database system
just to load and build the indices. Then at least another day should be
alloted to allow the database vendor to run the test, and make "reasonable"
tuning adjustments. ("reasonable" presumably does not include making
modifications to the database code itself, but does include adjusting
buffering, data layout on spindles, etc.) Sounds like something of
a logistical nightmare.

	I really do wish Neal Nelson luck with this effort, but I'm
just not sure that it can be pulled off at a trade show. On the brighter
side, there are apparently efforts underway (some reported on and talked
about at SIGMOD) to try and come up with more meaningful benchmark tests
which are better representations of the "real" world. (What, you
think the average OLTP application does 60% updates to 40% retrievals
like TP1 does?) Also, the conditions under which the benchmarks are
to be run are being more strictly defined, to try and reduce the
apples and oranges comparisons. So maybe in the next year or so we
will see more definitive benchmarks being run and published.