Path: utzoo!attcan!uunet!wuarchive!brutus.cs.uiuc.edu!tut.cis.ohio-state.edu!mailrus!ames!uhccux!munnari.oz.au!murtoa.cs.mu.oz.au!ditmela!yarra!chris
From: chris@yarra.oz.au (Chris Jankowski)
Newsgroups: comp.arch
Subject: Re: Memory utilization & inter-process contention
Keywords: paging, swapping, scheduling
Message-ID: <4056@yarra.oz.au>
Date: 24 Aug 89 06:45:07 GMT
References: <3332@blake.acs.washington.edu>
Reply-To: chris@yarra.oz.au (Chris Jankowski)
Organization: Pyramid Technology Australia, Melbourne
Lines: 51

In article <3332@blake.acs.washington.edu> lgy@newton.phys.washington.edu (Laurence Yaffe) writes:
>     As a concrete example (typical of what I've been trying to run lately),
> suppose you have a 32 Mbyte system and that there are only two processes
> running, each of which uses 25 Mbytes of virtual memory (almost all data space)
> with rapid, random access patterns.  (I.e., normal behavior for a program like
> Mathematica.)  If only one job were present, it would fit in real memory and
> the machine would run at virtually 100% cpu utilization.  With two such jobs
> in a demand-paged environment what appears to happen is that each process
> typically gets enough real memory so that it denies the other process 
> sufficient memory to run effectively - i.e., both processes continually
> page fault and cpu utilization plumments.
>
>     This sort of contention could obviously be prevented by a smarter
> scheduling technique - something which would entirely swap out one
> process for a period of time in order to let the other process run.
>
> Questions:
>
  .....
>     How common is this type of problem?  Anyone else share my feeling that
> many Unix systems could do a lot better at handling large processes?
> Anyone else share my feeling that this is important (and sadly neglected)?

I believe that the main design objective for UNIX scheduler was to achieve
good *response* time for interactive users and it does that unless there is
a severe resource depletion in the system due to excesive load.

In your particular problem you seem to try to maximise *throughput*.
There is an obvious tradeoff here and there are no free lunches.
The best way to do it is to create a batch enviroment in the 1960s mainframe
style. UNIX System V has a ready mechanism for you to run a single
stream batch queue. There are also many third party products which implement
fancy multistream batch queues with even fancier management.

I think that no scheduler can solve the problem in real life situation
except for such an artificially simple example. In reality you would need 
the scheduler to predict all *future* requirements of all processes.
Even if the processes could signal its future requirements to the scheduler
resolution will require a lot of processing time and at the current level
of technology not many people can afford one of their processors to spend
all its time calculating future savings. But the processes do not know
their future requirements either.


      -m-------   Chris Jankowski - Senior Systems Engineer chris@yarra.oz{.au}
    ---mmm-----   Pyramid  Technology  Australia	    fax  +61 3 820 0536
  -----mmmmm---   11th Floor, 14 Queens Road                tel. +61 3 820 0711
-------mmmmmmm-   Melbourne, Victoria, 3004       AUSTRALIA       (03) 820 0711

"Knowing how things work is the basis for appreciation,
and is thus a source of civilized delight."  -- William Safire