Xref: utzoo comp.arch:11133 comp.sys.mips:119
Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!gem.mps.ohio-state.edu!rpi!crdgw1!sunroof!hammondr
From: hammondr@sunroof.crd.ge.com (richard a hammond)
Newsgroups: comp.arch,comp.sys.mips
Subject: Re: Memory utilization & inter-process contention
Message-ID: <1900@crdgw1.crd.ge.com>
Date: 25 Aug 89 12:18:06 GMT
References: <3332@blake.acs.washington.edu> <13773@polyslo.CalPoly.EDU>
Sender: news@crdgw1.crd.ge.com
Reply-To: hammondr@sunroof.crd.ge.com (richard a hammond)
Followup-To: comp.arch
Distribution: usa
Organization: General Electric Corp. R&D, Schenectady, NY
Lines: 44

lgy@newton.phys.washington.edu (Laurence Yaffe) writes:
>
>>...
>>Severe, and unnecessary, performance degradation can occur if you have several
>>processes which have very poor locality of reference when accessing their
>>data, are sufficiently small that any singly process can fit in real memory,
>>and yet sufficiently large that no two processes will fit in real memory.

In article <13773@polyslo.CalPoly.EDU> ttwang@polyslo.CalPoly.EDU (Thomas Wang) comments:

>To avoid excessive page fault rate, a good algorithm is to load in multiple
>pages depending on system load.  Actually how many pages should you load in
>is still a research question.  In HP's term, loading in extra pages is
>called 'pre-fetching'.
>
>HP's MPE/XL have pre-fetching.  I don't know about Unix, but my guess is
>that Unix probably does not.

Note that in the case postulated by L. Yaffe, pre-fetching makes the thrashing
more severe, because the memory accesses have "poor locality of reference",
i.e. the working set is very close to the total memory required by the
process.  Fetching pages n through n+m because of a page fault on n
would increase the I/O overhead without increasing the probability
that you would avoid the next page fault.

Historical commentary:
The early multi-user UNIX machines, DEC PDP 11/45 and 11/70, could support
more physical memory than the largest process could use, so the scheduler
didn't worry about this sort of thrashing.

VM works fine if you have good locality of reference, while it fails miserably
if you don't.  With poor locality of reference, you need to have enough
physical memory to keep the job in memory.

Practical commentary:
Yes, the problem is interesting and can be solved, but I never observed
it except on our large number crunching machines, i.e. the Convex and Alliant.
On those machines I scheduled the jobs by hand, after all, if the job is going
to take 10 days of CPU time, you can just adjust the jobs so one runs over
night and the other one runs during the day.  So, I conclude that the problem
isn't that interesting for the vast majority of UNIX systems, i.e. all
our Suns and Vaxen didn't have the problem.

Rich Hammond