Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!usc!elroy.jpl.nasa.gov!sdd.hp.com!hplabs!hpcc05!hpyhde1!hpycla!hpcuhc!spuhler From: spuhler@hpcuhc.cup.hp.com (Tom Spuhler) Newsgroups: comp.benchmarks Subject: Re: TPC-B - is this really progress? Message-ID: <115440006@hpcuhc.cup.hp.com> Date: 3 Apr 91 19:13:55 GMT References: Organization: GSY Systems Performance Section Lines: 38 / hpcuhc:comp.benchmarks / jonathan@cs.pitt.edu (Jonathan Eunice) / 6:56 pm Apr 1, 1991 / renglish@cello.hpl.hp.com (Bob English) writes: jonathan@cs.pitt.edu (Jonathan Eunice) writes: > I notice a number of vendors, such as Sun and (given a recent > comp.arch posting) DG, concentrating on TPC-B benchmarks, rather than > TPC-A. With all due respect to the TPC folks, who do appear to be > trying to make the world safer for benchmarking, isn't this the same > deal we had before with TP1 (the TPC-B precursor) benchmarks? The two measure different things. TPC-A numbers measure the actual performance you would get if you built a local automatic teller network with thousands of terminals. It's basic premise is that it doesn't make sense to claim 10,000 TPS, unless your system can support 100,000 users and unless you price it with 100,000 terminals and terminal connections. That is not necessarily what people use their systems for. In addition, TPC-A numbers are difficult to get. It takes many runs, using lots of equipment, to get an accurate assessment. For systems not intended for the marketplace described above, this level of effort is difficult to justify. --bob-- Yes, but what does TPC-B measure? What useful thing, that is? If TPC-A demands a scaled-up system that quasi-accurately reflects a realistic OLTP use ("the actual performance you would get if you built a local automatic teller network"), TPC-B measures what? A system that is *not* configured like one that you'd find in a real OLTP system? This is my impression--that TPC-B reflects a highly synthetic, highly unrealistic configuration unlikely to be found in real OLTP situations. Just like the TP1 rubish we've had for some time. If I'm wrong on this point, tell me why. Otherwise, tell me why TPC is promoting an OLTP benchmark that does not reflect anything resembling the reality of OLTP computing. ----------