Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!iuvax!rutgers!cs.utexas.edu!uunet!utoday!greenber From: greenber@utoday.UUCP (Ross M. Greenberg) Newsgroups: comp.unix.i386 Subject: Re: _UNIX_Today!_ hits a new benchmarking low Message-ID: <966@utoday.UUCP> Date: 2 Sep 89 05:03:00 GMT References: <16054@vail.ICO.ISC.COM> Reply-To: greenber@utoday.UUCP (Ross M. Greenberg) Organization: UNIX Today!, Manhasset, NY Lines: 63 This is me in an official capacity for UNIX Today!. First, in the future, please make sure to advise me of an ongoing discussion regarding UNIX Today: our objective is to get better at what we do. Sometimes we do a damned fine job. Sometimes we don't. Based on what I'm reading here, I guess we blew the benchmarks on the MPE. Main reason: we're small. We don't have all the hardware and software we need in-house to get all the benchmarking we'd like. We're addressing that most rapidly with some stuff I can't go into until the deal is signed. Suffice it to say that I hereby promise you that screw-ups as you believe happened will never happen again. But...how did it happen, you might wonder? Well, those benchmarks *were* done honestly. Nothing phoney baloney about it. They were not done to make the MPE look good -- or bad. Those are the real numbers that came out. It is difficult to benchmark a new piece of hardware. Frankly, I don't hold much faith in the stone tests, prefering an application simulation myself. But, getting your hands on a piece of hardware and then figuring the best benchmarks to run, and what to run them against is a tough thing to do on a two week production cycle. Not a good enough excuse, I know. Now, we use a lot of freelancers. Bob Morien, the author of the MPE review, happens to be one of them. Freelancers work *directly* under my supervision. No review or techie stuff gets out into the paper without my personal approval. I am the one ultimately responsible for the errors in judgement in letting those particular benchmarks get printed. Due to some heavy deadline pressures, *I* didn't review them as thouroughly as I normally would have. I blew it. I screwed up. I should have gone back to Bob and asked him to rerun the benchmarks with more current and comparable stuff. Bob's actually a very good benchmarker. He simply didn;t have all the hardware and software on-hand at the time to re-run the benchmarks. He screwed up a little. I screwed up in a major way. If it's any consolation to you, I noticed the problem when we got back the proofs. It was too late at that point to do anything about it as the paper was to be printed about two hours later. We're working hard to make the paper better. We need your help. If we screw up, send the mail directly to me, or advise me immediately what newsgroup to join to take part in the discussion. Anyway...we'll be reviewing quite a bit of hardware in the very near future. You can be sure that the benchmarks for that hardware will show a comparison with the MPE benchmark. If it's a dog, the readers will know. If it's a shining light, they'll know, too. Benchmarks are always a problem spot. In two or three issues, you'll see what we're doing about it. I'll bet you all a drink at the next UseNix that you'll be more than satisfied. But you have to promise me that, once I drink all those drinks I win, somebody helps carry me back to my room, ok? Ross M. Greenberg UNIX TODAY! 594 Third Avenue New York New York 10016 Review Editor Voice:(212)-889-6431 BBS:(212)-889-6438 uunet!utoday!greenber BIX: greenber MCI: greenber CIS: 72461,3212 To subscribe, send mail to circ@utoday with "Subject: Request"