Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!rutgers!mailrus!wuarchive!uunet!prowest!pan
From: pan@propress.com (Philip A. Naecker)
Newsgroups: comp.databases
Subject: Re: RDBMS benchmark data
Message-ID: <1749@propress.com>
Date: 17 Dec 89 19:01:44 GMT
References: <4703@uswat.UUCP>
Distribution: usa
Organization: Professional Press West Coast Offices, Pasadena, CA
Lines: 84

In article <4703@uswat.UUCP>, bruce@zeb.USWest.COM (Bruce W. Hoylman) writes:
> I'm evaluating RDBMS packages for potential use in a major
> development effort in my company.  I am having trouble
> locating *ANY* type of benchmark/performance data for
> the RDBMS packages I am considering.  Maybe I'm looking
> in all the wrong places.
>
>...
> Bruce W. Hoylman (303-889-5806)  -- Standard disclaimer: "These views are
> bruce@mirage.USWest.COM             probably my own ..." --
> ncar!boulder!uswat!bruce
--

Without intending to be insulting, a reasonable question is "Why do you think
that this information might be of help to you?"  As a consultant who
specializes in building high-performance data management applications (almost
always using a relational database underneath), I can tell you that "Your
mileage may vary" is more than just a cute saying in the area of relational
database performance - it is the critical factor that you must understand and
take to heart.

If you believe that performance is going to be a major factor in your selection
of an RDMS, then I strongly suggest that you use a suite of benchmarks built
from your own (most important) applications.  Anything else is just not
sufficiently meaningful to be considered in any rational sort of way.  At the
very least, to make use of a "standard" benchmark you would need a deep
understanding of the relationship between you own application(s) and the
standards.  And since the standards as generally completely unrealistic in a
great many ways, even if you have that understanding you will likely have very
many unanswered questions regarding how your own application will perform.

Of course, almost no one either has (1) a sufficiently complete understanding
of their most critical applications to build such a benchmark suite, (2) the
time, money or inclination to do so, or (3) the environment in which to perform
such tests.  Cie la vie.

If you happen to be lucky enough to have a narrow range of applications that
you want to use with the RDMS, i.e.:

	o	a very update-intensive application

	o	an application with mostly ad-hoc query to stable data

	o	a read-only application with full reloads of the database

	o	a single-writer application

	o	or some other rather monolithic characteristic that dominates
		all database accesses

then you might be able to use some standard benchmarks that represent such
extremes in normal RDMS usage.  However, most *real* applications are not
easily represented by such extremes.

If you haven't already done so, I suggest you stop spending time looking for
benchmarks and start spending time rigorously documenting the characteristics
of your application(s).  Such parameters as transaction volumes, a preliminary
data model, update characteristics, referential integrity requirements,
hardware constraints (especially in the disk subsystem), requirements for
distribution through the network,  de-normalizations you might be willing to
make for performance, and so on.  These factors will be of much more use to you
in constructing a truly useful benchmark (even if it is synthetic) than will
most of the information you are likely to find about the publically available
benchmarks and how they perform in your proposed hardware environment.

Of course, the above steps are useful not only for understanding the
performance impacts of your RDMS choice, but also for identifying differences
between RDMS's in other important areas (ease of development, tools for
performance monitoring, data integrity characteristics, portability, network
distribution, etc.)

Even in building high-performance systems, I believe that the performance of
the database system itself (given any of the major players, including some you
haven't listed) is probably no more than 10% of the problem.


_______________________________________________________________________________
Philip A. Naecker                            Consulting Software Engineer
Internet: pan@propress.com                   Suite 101
          uunet!prowest!pan                  1010 East Union Street
Voice:    +1 818 577 4820                    Pasadena, CA  91106-1756
FAX:      +1 818 577 0073                    Also: Technology Editor
                                                   DEC Professional Magazine
_______________________________________________________________________________