Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP Path: utzoo!utgpu!water!watmath!clyde!rutgers!ames!oliveb!felix!fritz!martin From: martin@fritz.UUCP Newsgroups: comp.arch Subject: Social Issues in Benchmarking Bake-offs Message-ID: <4667@fritz.UUCP> Date: Mon, 1-Jun-87 17:23:57 EDT Article-I.D.: fritz.4667 Posted: Mon Jun 1 17:23:57 1987 Date-Received: Thu, 4-Jun-87 02:49:34 EDT Reply-To: martin@felix.UUCP (Martin McKendry) Distribution: world Organization: FileNet Corp., Costa Mesa, CA Lines: 93 Gentlemen: I have read with interest the discussion on benchmarking. I am sure that a benchmarking bake-off would be a good thing for prospective customers. But for the engineers in the audience, I would like to point out some of the social problems associated with touting your company's wares. I, for one, was recently fired ("laid off") in a manner that would suggest it was associated with benchmarking activities (and that was internal to the company). So were my boss, people who worked for me (some made it out before the formal festivities), and an entire performance group. This, of course, is my personal opinion. The formal reason was job elimination due to merger. At the time we were working on performance evaluations & improvement proposals for a particular mainframe architecture. In article <4294@nsc.nsc.com>, grenley@nsc.nsc.com (George Grenley) writes: > May the best CPU win! In article <2128@hoptoad.uucp> gnu@hoptoad.uucp (John Gilmore) writes: > > Let's have the bake-off in the trade show at, say, next Winter > Usenix. Probably the actual setup and running of the benchmarks > can be done a day or two before the show, so the results can be > printed for distribution, and to give the losers time to think > up (and print up) good explanations before we descend on them :-). >... >-- >Copyright 1987 John Gilmore; you may redistribute only if your recipients may. Any bake-off will have one winner and many losers. As an engineer working for a particular vendor, you may have run tests that convince you that you will be a winner. But be assured that in some dimension you will be a loser. And if you proposed that your company enter, you will be the one "responsible" for the exposure of "weakness". There has been some discussion of "holistic" vs "reductionist" approaches to benchmarking. We went through this. We tried various tests of particular components. Mostly industry standards. We did finally get through with these, but were met with arguments that the particular way the system was put together was such that the sum of the whole exceeded the individual parts. The problem with this argument is that it is extremely difficult and expensive ($10**6+) to prove or disprove the conjecture in a manner that is "finally" convincing. And then, of course, any vendor can point to happy customers (forget dissonance) to show that rational people are buying the gear, so it must be better than the competition's. We found that management a) did not understand the notion of component (reductionist) tests, or the orthogonality (or lack theoreof) of particular components, and b) did not care to hear discussion. I will predict that any open bake-off will lead to problems for almost anyone who enters it. There are very few "clean wins" to be had. There are lots of arguments to be had -- was array bounds checking on? Was loop unrolling allowed? Does their equipment detect null-pointers in hardware? None of these arguments are understood by management (read that: "your management"), who are generally just hacked that the discussion is going on at all (particularly if its in the public domain). There are some companies that are likely to benefit hugely from such an event. They, of course, are the small companies, eager to get market share and with little market share to protect, who have been able to design an architecture to exploit current technology without regard for compatibility :-). For them, the likelihood of "winning" and the attendant publicity can do nothing but good. Who could they be? For myself, I'd love to see such an event, and since I'm not foolish enough to get involved, it won't hurt me. So I say go for it! Martin S. McKendry FileNet Corp {hplabs,trwrb}!felix!martin Disclaimer: The statements above are completely untrue. They are the author's alleged opinion only, and even he is not sure he would admit to them. They in no way reflect the opinion of the author's current employer or previous employer (especially previous employer). All claimed facts have already been discredited. -- Martin S. McKendry FileNet Corp {hplabs,trwrb}!felix!martin