Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!usc!cs.utexas.edu!uwm.edu!bionet!agate!ucbvax!mtxinu!sybase!ohday!tim
From: tim@ohday.sybase.com (Tim Wood)
Newsgroups: comp.databases
Subject: Re: sql benchmarking environment (maybe tpc-b?)?
Keywords: benchmarking sql sybase
Message-ID: <12423@sybase.sybase.com>
Date: 21 Apr 91 04:41:27 GMT
References: <3632@dagobah.UUCP>
Sender: news@Sybase.COM
Organization: Sybase, Inc.
Lines: 47

In article <3632@dagobah.UUCP> mis@Seiden.com (Mark Seiden) writes:
>
>i have so far been unsuccessful in persuading sun or sybase to part with
>their tpc-b implementation, though they seem very happy to quote numbers
>in press releases and semi-technical quasi-marketing literature.
>
>pity it isn't currently possible to replicate the benchmarks on other
>configurations than they ran it on.  (or am i misstating the case?)

A couple of inaccuracies: the TPC-B is a documented benchmark.  Anyone
is free to create the schema, emulate the workload on the desired 
configuration and measure the results.  

The whole idea of a fixed configuration is to control that variable in 
order to make a meaningful comparison between products for a standard 
workload.

The technology we use to emulate the workload is probably proprietary; 
after all, it saves labor to know how to model accurately x00 terminals
firing transactions at a database.  How would it be in a vendor's interest 
to tell everyone (ie competitors) how that's done?  Moreover, if a vendor
publishes an implementation, it incurs a support obligation.  That can
put it in the position of supporting the benchmarking efforts of those
with contrary interests, who may not exercise complete impartiality
before making performance claims about the product.  It is then very
difficult for the vendor to evaluate the measurer's methods.
(Mini-disclaimer: this statement is completely hypothetical, it does
not apply to any particular vendor or individual.)

The inflexibility you perceive is greater elsewhere in the industry;
there are companies that prevent, as a condition of the software licence,
those who run their own benchmark implementations from disclosing their 
measurements.

There are two kinds of useful benchmark: 1) audited ones of a standard 
workload on a constant configuration, used to comparison-shop among vendors 
and produce a short list; 2) the prospective customer's workload, or a model
of it.  (2) is the more important, because it predicts better
how the DBMS will perform on one's application; this is the only
important question, ultimately.

-TW
---
Sybase, Inc. / 6475 Christie Ave. / Emeryville, CA / 94608	  415-596-3500
WORK:tim@sybase.com     {pacbell,pyramid,sun,{uunet,ucbvax}!mtxinu}!sybase!tim
PLAY:axolotl!tim@toad.com		       {sun,uunet}!hoptoad!axolotl!tim
Dis claim er dat claim, what's da difference?  I'm da one doin da talkin' hea.