Path: utzoo!attcan!uunet!lll-winken!lll-lcc!pyramid!infmx!aland From: aland@infmx.UUCP (Dr. Scump) Newsgroups: comp.databases Subject: Re: ORACLE on the cheap... questions (LONG) Summary: Summary of the Oracle/RTI/Informix benchmark discussion Keywords: Joe Isuzu II benchmark wars hypocracy con Message-ID: <302@infmx.UUCP> Date: 19 Jul 88 10:09:16 GMT References: <5165@dasys1.UUCP> <8208@ncoast.UUCP> <178@turbo.oracle.UUCP> <180@turbo.oracle.UUCP> Organization: Informix Software Inc., Menlo Park, CA. Lines: 242 : Brandon Allbery: | Oracle has rollback, rollforward, and transaction logs. The version I used | (5.1) wasn't too hot on the speed front (watch, however, for Turbo Oracle) : Robert Bradbury, Oracle: | From rbradbur@oracle.UUCP@munnari.oz Fri Jul 8 16:19:35 1988 | Date: 8 Jul 88 23:19:35 GMT | References: <5165@dasys1.UUCP> <8208@ncoast.UUCP> | Organization: ORACLE Corporation, Belmont CA, USA | | I *hate* statements like "wasn't too hot on the speed front". Exactly | *what* are you doing that gives you that impression? We beat Informix | and Ingres in a good percentage of the DeWitt benchmarks on a number of | machines. The functionality of a full-function RDBMS (transaction logs, | security, rollback, roll-forward, read-consistancy, etc.) imposes a | certian amount of overhead. You cannot compare the performance of | database systems which have those features with those that do not. So, what is the implication here -- that the other products lack these features? How about some support for this claim? What is a "good percentage of" DeWitts? How many is "a number of machines" (and out of how many in the overall sample)? : Dave Kellogg, Relational Tech: | From: davek@rtech.rtech.com (Dave Kellogg) | Newsgroups: comp.databases | Subject: Re: ORACLE on the cheap... questions | Date: 13 Jul 88 03:33:41 GMT | References: <5165@dasys1.UUCP> <8208@ncoast.UUCP> <178@turbo.oracle.UUCP> | (Excellent comment on standards and benchmarks and their | pitfalls omitted) | | | A Note On "The Net" | ------------------- | | This "WE beat vendor X" type quote, however, disturbs me. If this continues, | and becomes standard operating procedure for this newsgroup, it could easily | mean that comp.databases will degenerate into nothing more than a forum for | vendor cross-fire. This degeneration is not unfeasible given that companies | like Relational Technology, Informix, and Oracle employ literally thousands of | people, and each could easily place a designated "net monitor" to fire back | accusations at the other vendors. | A little inter-vendor "teasing" is probably inevitable, but that's where | I think it should stop. Cluttering the net with "WE beat so-and-so" will | neither reward the readers of this group, nor, in actualitly, any given | DBMS vendor. | | David Kellogg Amen. : G Pavlov, Harvard U. | From: pavlov@hscfvax.harvard.edu (G.Pavlov) | Subject: Re: ORACLE on the cheap... questions | Date: 14 Jul 88 15:40:55 GMT | References: <5165@dasys1.UUCP> <8208@ncoast.UUCP> <178@turbo.oracle.UUCP> | Organization: Health Sciences Computing Facility, Harvard University | | The only independent comparison benchmark (Ingres, Informix, and Oracle) | was a rather extensive one performed by a Palmer & Associates, Inc. | (Duluth, Ga). This was performed on an IBM AT and involved apx. 130 sep- | arate query or update processes. Several lines from the "executive sum- | mary" of the report: | | "In general (using a geometric mean of normalized data approach) Ingres | was from 1.63 to 2.86 times faster than Informix, and Ingres was from | 10.56 to 29.00 times faster than Oracle." | | "Oracle was clearly outdistanced by both Ingres and Informix with 5 first | place, 40 second place and 65 third place finishes." (in retrievals) | "Oracle won 4 of 20 update record queries." | | - etc. | | This is not to say that this is the definitive word on these dbms's. But I | have to give it more credence than the "results" that have come from the | individual vendors themselves. | : Stephen Deal, Kodak | From: deal@kodak.UUCP (Stephen M. Deal) | Subject: Re: ORACLE on the cheap... questions | Date: 16 Jul 88 01:58:01 GMT | References: <5165@dasys1.UUCP> <8208@ncoast.UUCP> <178@turbo.oracle.UUCP> <590@hscfvax.harvard.edu> | Organization: Eastman Kodak Co, Rochester, NY | | In my book, benchmarks aren't worth anything unless: | a) All the products are run on the SAME machine. | b) Each product runs the same set of transactions. | c) Each vendor is permitted to tune their own product. I would add d) Or, NO vendor is allowed to do application- or benchmark- specific testing (as the user would likely experience in a real-world situation) and e) Similar products are compared (e.g. compiled vs. compiled, interpreted vs. interpreted, closest match of capabilities) Now, for the fun part. : Robert Bradbury, Oracle | From: rbradbur@oracle.UUCP (Robert Bradbury) | Subject: Re: ORACLE on the cheap... questions | Summary: Benchmarks and environmental information | Date: 16 Jul 88 04:25:26 GMT | References: <5165@dasys1.UUCP> <8208@ncoast.UUCP> <178@turbo.oracle.UUCP> <590@hscfvax.harvard.edu> | Organization: ORACLE Corporation, Belmont CA, USA | | In article <590@hscfvax.harvard.edu>, pavlov@hscfvax.harvard.edu (G.Pavlov) writes: | > In article <178@turbo.oracle.UUCP>, rbradbur@oracle.UUCP (Robert Bradbury) writes (in response to another article): | > > | > > | > A while ago I attempted to arrange a set of cooperative benchmark runs | > with other sites running other dbms's. The two Oracle sites I talked to | > claimed that their license agreements forbade publication of benchmark | > results. Don't know if this is completely true, but that is what I was | > told. | | This is true. The license agreement prohibits publication of benchmark | results. One reason for this is that some vendors will publish results | comparing apples with oranges and not make it completely clear to the user. NO KIDDING!!! Did you happen to miss the ads Oracle ran in UNIX WORLD for the better part of the year "comparing" Oracle and Informix? | ... | Yes, YOUR RDBMS dollars are paying lawyers to play these games. sigh. Come on! Just who started this benchmark "game", anyway? | > (Quotes the Palmer & Associates benchmarks again) | | This is *EXACTLY* the type of thing I was refering to in my original | posting. You *fail* to mention the versions which were being compared | in this benchmark. Oracle, Informix and Ingres have had a history of | leapfrogging each other in performance benchmarks over the last 5 years. | If you take production releases at one point in time you get one set of | results; at another point in time you get another set. You don't mention | what operating system the benchmark was performed on, what the system | configuration was, how much time was spent tuning the systems, etc. | And of course numbers from a PC AT are generalizable to VAXes, Suns, | Sequents, etc. :-) NO KIDDING! I think you will have a tough time defending this position to anyone who has seen your ads in UNIX WORLD. (For example, the June '88 issue, page 10). This ad claims to depict a "benchmark" in which Oracle claims a resounding performance victory over Informix. Some highlights: 1) regarding your gripe about "you *fail* to mention what versions were being compared in the benchmark": in this ad, you run Oracle 5.0 (current version) against Informix version 2.0. Version 2.0 is OVER TWO YEARS OLD. The current version for OVER A YEAR before this ad ran was 2.10.00 (now 2.10.03 is current). There were also 3 other releases in between, including a major enhancement release (2.00.05) which came out in December *86*. 2) Informix-TURBO is displayed in the ad photo, implying that the "benchmarks" were run against TURBO, which was NOT TRUE. TURBO did not exist in the version listed as benchmarked. (I just dare ya to let an independent run a legit benchmark of Oracle vs. Turbo). 3) The ad states that the benchmark is an "industry standard" benchmark, but DOES NOT NAME IT. It doesn't even name the PRODUCTS used; it could have been compiled embedded SQL against an interpreted ACE report, for all the ad says. 4) The ad photo depicts a 3B2, although the benchmark was supposedly run on a Sun, implying that the performance stats apply to their "fire sale" 3B2 port. | ... | As an example of how poor benchmarking efforts are I'll make the | statement that no one who has ever benchmarked RDBMS systems on | UNIX has ever bothered to verify which RDBMS really guarantee that | when you commit your transaction the data is physically on the disk. | (I'm sure I'll get some comments on this - :-)). I make that statement | because in 5 years of benchmarking comparisons no one has ever told me | they knew how to monitor system calls or go in and examine the UNIX | FILE table to determine blocks were being written through the cache to the | disk. The best they usually do is unplug the machine to see if the database | got corrupted -- sorry folks but that doesn't provide *proof* unless | you do it a few thousand times when you are executing 30 transactions | a second. --------------- -------- Won't touch that one... | | Oracle in fact lost a number of benchmarks over the years because | other vendors were ignoring the issue of data integrity. Ah, now to the new ad campaign. Integrity. You forget that UNIX from the outset handled writes asynchronously for cooked devices -- I doubt that any vendor really "ignored" this situation. This is part of the reason why several vendors came out with raw i/o capability, I suppose. Recently, some versions support forced writes (via the O_SYNC flag in System V and O_WSYNCH in XENIX), but this is new. The O_SYNC flag doesn't even show up in AT&T's printed System V Programmer's Manual Volume 2 (? - I'm at home, I think it's volume 2. The one on system calls.) My copies, anyway :-]. Besides, the real issue in the transaction logging methods with which I am familiar (4 vendors encompassing UNIX, MVS, and VM, not including Oracle, so this may not apply to Oracle) is whether the write completed to the *transaction log*, not the database. In the implementations I know, the data on disk is essentially trashed by the recovery process; the most recent backup is copied over the data and all committed transactions are reprocessed against the backup, beginning from the backup or a checkpoint. (Or, the transactions after the checkpoint are removed, then reprocessed from the log in one other implementation; I never did really understand that one). The funniest thing is that they choose the Suns to run their benchmarks. Sun 3.X machines DO NOT SUPPORT synchronous writes anyway (no O_SYNC flag here, folks), so any claim that their benchmarks are hurt by "integrity issues" on these machines is BOGUS. The only way to force i/o is to use raw devices; Oracle decries raw devices as being "complex" in their current ad. (If my understanding of Sun 3.X synchronicity is wrong, I will post a followup and apology. I confirmed this with a friend at Sun about a month ago). The current ads do *not* indicate that this integrity claim applies only to certain (e.g SVR2+) ports, that I recall. | Sorry this turned out so long but I like to try and present a balanced | view of things. Oh? :-] Mine also turned out longer that I expected or wanted, but I wanted to present another side. Now that everyone has been heard from, can we take any further discussion to email? Thanks. I speak strictly for myself and not Informix; I am solely responsible for any inaccuracies. | Robert Bradbury | Oracle Corporation | (206) 784-9474 hplabs!oracle!rbradbur -- Alan S. Denney | Informix Software, Inc. | {pyramid|uunet}!infmx!aland Disclaimer: These opinions are mine alone. If I am caught or killed, the secretary will disavow any knowledge of my actions. Santos' 4th Law: "Anything worth fighting for is worth fighting *dirty* for"