Path: utzoo!attcan!uunet!husc6!mailrus!ames!lll-tis!daitc!jkrueger@daitc.daitc.mil
From: jkrueger@daitc.daitc.mil (Jonathan Krueger)
Newsgroups: comp.databases
Subject: Re: Oracle Peformance Simulation
Message-ID: <246@daitc.daitc.mil>
Date: 22 Nov 88 16:58:26 GMT
References: <3352@newton.praxis.co.uk> <557@4gl.UUCP>
Sender: jkrueger@daitc.daitc.mil
Reply-To: jkrueger@daitc.daitc.mil (Jonathan Krueger)
Organization: Defense Applied Information Technology Center
Lines: 71
In-reply-to: honzo@4gl.UUCP (Honzo Svasek)

In article <557@4gl.UUCP>, honzo@4gl (Honzo Svasek) writes:
>From article <3352@newton.praxis.co.uk>, by ben@praxis.co.uk (Ben Dillistone):
>> I am in the process of trying to create a performance evaluation
>> environment for an interactive/batch Oracle system.
>. . .
>> The problem is simulating variable numbers of users.
>> The only solution so far is to use a script repeatedly running SQL*Forms
>> with an pre-created echo file generating the key strokes. A single such
>> script generates about 150 queries a minute on a MicroVAX (can you type
>> that fast?).
>
>I have worked on a simular simulation for Unify/Accell on Unisys hardware.
>We started of the way you describe. It gave some idea of the speed of the
>machine for this application, but was not really realistic.
>
>After quite some experimenting, we ended up with a system that used pty's
>and some sort of feedback.
>
>Two programs talking to the master end of the pty. A writer program (the
>hands of the typist) and a reader program (the eyes of the typist).

I am currently writing up the results of a similar benchmark performed
here.  The advantages of this type of testing include:

	measures specific software and hardware of interest

	yields clear, unambiguous results

	predicts real loads from real users under real conditions

	finds limits of acceptable performance with parametric testing

	finds system, application bottlenecks, identifies where to invest
	system upgrades or application optimization

In other words, it's highly empirical and directly relevant.  But the
disadvantages include:

	impossible to be sure how well simulations predict actual users,
	but accuracy of prediction depends entirely on this guess

	results are specific to exact hardware config and environment;
	no way to predict results on different processor, memory size,
	disk speed, operating system, application itself, version of DBMS

	highly human-intensive, therefore these data are expensive to
	collect, yet due to above must be recollected after each
	significant change to hardware, OS, application, or DBMS.
	Worse yet, definition of `significant' is circular; without
	reference to a model, only way to know what's significant is
	to re-run the simulation.

	Testing must be done on an unloaded system from driver
	software executing on another system, else artifacts render
	results unreliable; not only is this inconvenient, as it
	removes two shared systems from general availability, but it
	also tells you little about mixed workloads, i.e. what
	response database users will get when other users are active.

In other words, it's not a powerful method.  The ratio of data
collected to predictions generated is low.  Underlying problem seems
to be you're observing performance but not modeling it.

This topic comes up every now and again on comp.arch. I recommend John
Mashey's articles on performance measurement there, a recent summary
by Henry Spencer is <1988Nov18.200914.4636@utzoo.uucp>.  They warn
against small, synthetic benchmarks, single figures of merit, and
measures of performance which poorly represent real applications.

-- Jon
--