Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP Path: utzoo!utgpu!water!watmath!clyde!rutgers!labrea!decwrl!pyramid!prls!mips!mash From: mash@mips.UUCP Newsgroups: comp.arch Subject: Re: Benchmark results using SPICE Message-ID: <690@winchester.UUCP> Date: Thu, 17-Sep-87 00:56:51 EDT Article-I.D.: winchest.690 Posted: Thu Sep 17 00:56:51 1987 Date-Received: Sat, 19-Sep-87 09:21:06 EDT References: <677@obiwan.UUCP> <5002@columbia.edu> Reply-To: mash@winchester.UUCP (John Mashey) Organization: MIPS Computer Systems, Sunnyvale, CA Lines: 108 Keywords: (circuit simulator pgm) In article <5002@columbia.edu> yoram@cheshire.columbia.edu (Yoram Eisenstadter) writes: >In article <677@obiwan.UUCP> mark@mips.UUCP (Mark G. Johnson) writes: >>We finally located a VMS-VAX 11/780 with SPICE available, so we ran >>the three public domain SPICE benchmark circuits... >> Berkeley-2G6 >>MACHINE seconds Vaxes OS, compiler, notes >>------- ------ ----- ------------------------- >>VAX 11/780 1354.0 0.60 4.3BSD, BerkF77 V2.0 >>Microvax-II 993.5 0.81 Ultrix 1.1, fortrel >>SUN 3/160 901.9 0.90 SunOS 3.2 f77 -O -f68881 >>VAX 11/780 848.0 0.95 4.3BSD, fortrel >>VAX 11/780 808.1 1.0 VMS 4.4, fortran/opt >>SUN 3/260 744.8 1.1 SunOS 3.2 f77 -O -f68881 >>SUN 3/160 506.5 1.6 SunOS 3.2 f77 -O -ffpa >>SUN 3/260 361.2 2.2 SunOS 3.2 f77 -O -ffpa >>SUN 4/260 225.9 3.6 SunOS 4-beta2 f77 -O3 -Qoption as -Ff0 >>MIPS M/800 136.5 5.9 UMIPS-BSD V2.01, f77 V1.21 >>AMDAHL 470 V/7 125.5 6.4 VMSP-CMS 4.0, FORTVS 4.1 >>MIPS M/1000 114.3 7.1 UMIPS-BSD V2.01, f77 V1.21 >>FPS 20/64 48.0 16.8 VSPICE (2g6 derivative) >Is anyone besides me bothered by the choice of machines used in >this comparison? If you've followed this whole effort by Mark and co (with good cooperation from Dave Hough in particular & various folks elsewhere), the goal was to get a widely available set of good Spice benchmarks. To this end: Mark proposed some on the net, and asked for more examples. Collected public--domainable versison of ours plus others. Sent them to other knowledgable people for critiquing. Published the input decks, and asked for results. Published the numbers we've measured directly, or that others have been kind enough to send. This is a lot of work, and is not even easy to make happen: Spice breaks compilers on many machines, so even being able to run it says something useful. Everybody would be happy to see more numbers, and we often do run around begging for numbers from other machines. However, Mark does have to spend some time designing chips... We certainly believe in benchmarking as strong as machines as we can lay our hands on, but for some bigger machines we have no choice but to rely on others. Certainly this whole process as been as open as one can imagine, and is still early in sequence of doing it. Of the machines above: the 4.3BSD and MicroVAX our ours, as are the MIPS machines. Sun numbers were dgh's. I don't know where Mark got the others. > >Note that the 11/780 is a mid-1970's design using mid-1970's >technology (they were already being produced in '79 I believe). Yes. Fortunately, it's still a good metric in that it's familiar, and, after a while, you find fairly consistent performance ratios amongst VAXen. > >I think (I'm not so sure on this one) that the Amdahl 470 V/7 is >also one of Amdahl's earliest machines. (Does anybody know the >vintage of the FPS 20/64? Is it modern?) V/7 is old. > >I'd like to see benchmarks against some modern-day VAXes, e.g., >the VAX 8800, and against some modern-day big mainframes, e.g., We would, too! That's why Mark went to the pain of doing all this, since we can't quite afford either of those. >IBM 3090-class machines. Then, we might get some idea of how >little machines like the MIPS and SUN fare against current big >timesharing systems. (My guess is that the benchmark will run in >under 30 seconds on a 3090, and that the big VAXes will beat the >MIPS/1000...) Under 30 seconds: probably right, although maybe just barely. The only 3090 numbers I've got handy are for Hspice case "ST230", and taking the ratios versus both 11/780 and M/1000, you'd get either 28X faster or a bit less than 4X faster, so somewhere between 27 and 47 seconds seems right. I'd guess an 8700 [1 of the 2 cpus in an 8800] would do around 135-150 seconds. M/800s almost always beat 8700s on floating-point crunchers, and an M/1000 is 1.2X an M/800 (scales directly with clock on CPU-bound jobs), so that it is typically 1.4X-1.6X as fast as an 8700 (VMS), (i.e. halfway betweeen an 8700 and 8800 in thruput on these things). This kind of ratio has been consistently found with other programs, such as: Doduc (8700VMS = 5.2X 11/780, M/1000 = 8.5X) Digital Review benchmarks (8700 = 1.469 seconds geometric mean, M/1000 = .98) Livermore Loops (geometric mean, 8700 = 1.0MFlops, M/1000 = 1.65) Various Hspice jobs. > >Also, notably absent is the figure for the MicroVAX II running >VMS (which is what it was intended to run). What is the >"fortrel" compiler? Our MicroVax II runs Ultrix. We'd be glad to hear VMS numbers. Fortrel is a LLNL-derived VAX compiler with pretty reasonable optimization. We use it inhouse, especially if trying to approximate VMS numbers when we can't get them. As can be seen, it's about .95X performance relative to VMS, on the 780, and from othere experience, correlates not too badly. (The 3 11/780 numbers give you an idea of relative compiler performance, obviously important in this case.) Conclusions: 1) There's a pretty strong effort to get as good a set of numbers as possible. At least a few people have been quite cooperative, and nobody is trying to hide anything. 2) Send results to mark if you have some. 3) IF you have no results, but you have spare 8700s or 3090s around, send them to ME. I'll figure out something to do with them. Alternatively, if you have 3090s and 8700s lying around, send them, too. -- -john mashey DISCLAIMER: UUCP: {decvax,ucbvax,ihnp4}!decwrl!mips!mash OR mash@mips.com DDD: 408-991-0253 or 408-720-1700, x253 USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086