Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!usc!elroy.jpl.nasa.gov!sdd.hp.com!hplabs!hpcc05!hpyhde1!hpycla!hpcuhc!spuhler From: spuhler@hpcuhc.cup.hp.com (Tom Spuhler) Newsgroups: comp.benchmarks Subject: Re: TPC-B - is this really progress? Message-ID: <115440007@hpcuhc.cup.hp.com> Date: 3 Apr 91 19:58:29 GMT References: Organization: GSY Systems Performance Section Lines: 45 I think that my previous post got out (nasty old notes!!) but what I wanted to comment on was: TPC-A is a rigorous (as in well-defined) interactive database benchmark (TPC-B is a rigorour batch DBMS benchmark). They never have been thought of as particulalry indicative benchmarks, rather, they are intended to replace Debit/Credit TP1 as "the" DBMS benchmarks, in such a way so at least everyone would know exactly what was meant when someone reported "TPC-A" or "TPC-B" results. While the applicability of the results to a specific application may be weak, at least you have some assurance of the conditions under which the results were obtained, and can compare different machines performance and cost of solution (for the TPC workload). In this, I think that TPC has done very well. Only a few oddballs still bother to report TP1 or debit/credit results (and they are scorned), and there is significant peer review of results to insure compliance. The specification is detailed enough and the required reporting detailed enough that it it posible (IMHO) to make an apples to apples comparison of TPC numbers (A-A, B-B, as defined by TPC) and feel relatively comfortable with what you are seeing. This has NOT BEEN POSSIBLE in the past. With the earlier "benchmarks" what was actually done was largely a related to the vendors interprettion and ethics, with TPC, at least you know what you got. Now, anyone who is silly enough to report, say TP1 instead of TPC-B results, for example, is seen as trying to hide something. As to how useful these are in terms of customer application performance prediction, probably not all that much, HOWEVER, they are certainly better for anticipating the likely performance of a OLTP/DBMS application then any other common benchmark out there (Try it with Dhrystones!). There are various caveats, but now at least everyone is playing on the same field. Now, if that was the end of it, there might be a problem. However, things are just getting started in TPC land. Whereas TPC-[AB] were intended as stopgap efforts, TPC is finalizing it's first application oriented benchmark, TPC-C. I encourage everyone out there to get ahold of the spec, and read it. And they have more planned. [this is actually a nuisance in some ways as that means we will have to implement them, but this is in the total scheme of things: GOOD]. -Tom "TPC-A, TPC-B: ask for them by name"