Xref: utzoo comp.arch:11072 comp.sys.mips:92
Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!ncar!unmvax!ogccse!blake!lgy
From: lgy@blake.acs.washington.edu (Laurence Yaffe)
Newsgroups: comp.arch,comp.sys.mips
Subject: Re: Memory utilization & inter-process contention
Message-ID: <3342@blake.acs.washington.edu>
Date: 22 Aug 89 20:14:42 GMT
References: <3332@blake.acs.washington.edu> <1989Aug22.163100.25540@utzoo.uucp>
Reply-To: lgy@blake.acs.washington.edu (Laurence Yaffe)
Organization: University of Washington, Seattle
Lines: 52

In article <1989Aug22.163100.25540@utzoo.uucp> henry@utzoo.uucp (Henry Spencer) writes:
>In article <3332@blake.acs.washington.edu> lgy@newton.phys.washington.edu (Laurence Yaffe) writes:
>>suppose you have a 32 Mbyte system and that there are only two processes
>>running, each of which uses 25 Mbytes of virtual memory (almost all data space)
>>with rapid, random access patterns...
>>    This sort of contention could obviously be prevented by a smarter
>>scheduling technique - something which would entirely swap out one
>>process for a period of time in order to let the other process run.
>
>Have you ever figured out how long it takes to swap out a 25MB process?
>Or to swap it back in again?  The fact is, this example system simply
>does not have enough main memory for its workload.  The fix is, either
>double the memory or run the jobs in sequence rather than simultaneously.
>
>V7 /bin/mail source: 554 lines.|     Henry Spencer at U of Toronto Zoology
>1989 X.400 specs: 2200+ pages. | uunet!attcan!utzoo!henry henry@zoo.toronto.edu

    This response ("buy more memory, or sequence jobs by hand"), while common,
completely misses the point.  The real issue here is how to maximize overall
performance with a given physical configuration.  The problem I raised is
"scale invariant" - no matter how much memory one has, there will always be 
important jobs which fill all of memory.  A bigger memory just means that
I'll try to run bigger jobs.  This is why I deliberately phrased the initial
description of this problem in terms of ratios of process size to physical
memory.  Part of intended purpose of my posting was to emphasize that 
system performance in the presence of large processes (large relative to
however much memory you can afford, but not so large that the system
shouldn't be able to run them efficiently) IS important!

    As to your specific question - yes, I have considered how long it takes
to swap out a 25 Mb process.  On my system, perhaps 10-20 seconds.  This is
utterly negligible for jobs which require many cpu-hours or cpu-days.  
The real issue here is also "scale invariant": as process size grows
certainly the time to swap the entire process in or out grows.  But so
does the time required to page the same amount of data in from disk
and, I submit, so does the typical cpu time.  I see no reason to frown
on swapping strategies provided that the frequency of process swaps is
intelligently adjusted as a function of process size.

    Finally, while running multiple large jobs in sequence is a valid
suggestion and will certainly avoid inter-process contention, to me
this is an admission of failure.  (Why do we like Unix?  Multiprocessing?)
If a single person is starting multiple large jobs, then scheduling by hand
may be acceptable.  If different users are trying to run big jobs then
strict sequential scheduling is a real pain.  One should be able to do better.
While I may have high expectations, I do think that minimizing the type of
contention I discussed is a reasonable user desire and should be a relevant
operating system design goal.

-- 
Laurence G. Yaffe		Internet: lgy@newton.phys.washington.edu
University of Washington	  Bitnet: yaffe@uwaphast.bitnet