Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!csd4.csd.uwm.edu!gem.mps.ohio-state.edu!ginosko!uunet!utoday!greenber From: greenber@utoday.UUCP (Ross M. Greenberg) Newsgroups: comp.unix.i386 Subject: Re: _UNIX_Today!_ Message-ID: <971@utoday.UUCP> Date: 7 Sep 89 00:52:01 GMT References: <16054@vail.ICO.ISC.COM> <966@utoday.UUCP> <9468@chinet.chi.il.us> <357@simon.UUCP> Reply-To: greenber@utoday.UUCP (Ross M. Greenberg) Organization: UNIX Today!, Manhasset, NY Lines: 84 In article <357@simon.UUCP> steve@simon.UUCP (Steven E. Piette) writes: > We all make mistakes from time to time, but too many >hide from the heat when that time comes. > >Thanks Ross for being honest with us. > I figure it's easier to be honest than to try to coverup a mistake. I must admit that the problems the MPE review caused have already changed things here. The review coming out in the next issue is on the ATT 6386E unit. The benchmarks we *were* going to run were going to be be Whetstone/Dhrystone/IOstone on it, with 1 to five processes running. Without a comparison, they'd be almost useless, right? So, we went to do a comparative benchmark. The only machine we had available was a Compaq/386/16MHz running XENIX. It would have been obvious to anyone reading the review and seeing the benchmarks that we were obviously coparing apples and oranges (the 6386E is a 20Megger), but due to the postings in *this* group, I've opted to not include the benchmarks as stated. Instead, when we can more reliably cross-check the hardware and software (6386E & SVR3.2 v 6386E & XENIX v COMPAQ 386/20 & SVR3.2 v COMPAQ 386/20 & XENIX), then I'll include the benchmark data. But, that brings up some interesting points. As I've indicated, we're going to be doing some massive work in the benchmarking area shortly. Part of my responsibility is to figure out what those benchmarks really need, and how to compare apples and oranges. In the DOS world, it's easy. In the UNIX world, not so easy. So. I figure I'll ask the same guys that gave me such a hard time (thanks for volunteering! :-) ): what do *you* think is important in a UNIX benchmark? How do I compare Machine A (with hardware and software configuration of 'A') to Machine B with its H/W configuration of 'B'? Doe Whetstone/Dhrystones/IOStones cover enough? Application testing is important, obviously, but should it be *real* applications (limiting the number of machines we can test) or psuedo-apps (take a 100,000 x 1024 byte file and sort it a coupla hundred times while adding up a given field)? Is it important to test with one or more processes doing the same thing? How about with one or more *users* doing the same thing? Should the results be output in tabular format, graphical format, both? Should we list all tested machines in each issue (doubtful: each editorial page is precious)? Should a given machine's configured price be used to figure out which other machines to compare its benchmark to? And, which price? The list price or the quantity 100 price? Setting up a benchmarking suite is going to be an ongoing process for us. I'd like to ask your help in helping us to figure oput what to do. No promises, though: if we can't afford it in money or in time, we can't implement it. But, I *know* that I'm not a benchmarking expert! :-) I'd be happy to collect mailgrams with people's ideas, but an open discussion might prove a bit more interesting to all. One thing: flames regarding our *current* testing (or even compliments) should be sent to me at the below address. Until we get a biz.group set up, let me not use the net to tell people what a great magazine we are? :-) Anyway...just to lay one thing to rest. Space permitting, we'll be printing the MPE benchmarks versus a *current* version of Interactive as soon as possible. And the cartoon on our /etc page will probably show me wiping a bunch of egg off my face. >Now if I only could figure out what happened to the subscription I had... >(only two copies ever showed up here from the beginning of Unix Today!) > Sigh. Considering that our circ department is just now geting on-line, bear with us. We're getting there. Chances are you lost out when we requalified. Send mail to the below address and qualify again? Ross -- Ross M. Greenberg, Review Editor, UNIX Today! greenber@utoday.UUCP 594 Third Avenue, New York, New York, 10016 Voice:(212)-889-6431 BIX: greenber MCI: greenber CIS: 72461,3212 To subscribe, send mail to circ@utoday.UUCP with "Subject: Request"