Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!seismo!cmcl2!lanl!jlg
From: jlg@lanl.ARPA (Jim Giles)
Newsgroups: net.arch
Subject: Re: VERY LARGE main memories
Message-ID: <7839@lanl.ARPA>
Date: Tue, 23-Sep-86 19:59:53 EDT
Article-I.D.: lanl.7839
Posted: Tue Sep 23 19:59:53 1986
Date-Received: Wed, 24-Sep-86 22:02:28 EDT
References: <1130@bu-cs.bu-cs.BU.EDU> <7144@lanl.ARPA> <7148@lanl.ARPA> <609@nike.UUCP>
Reply-To: jlg@a.UUCP (Jim Giles)
Organization: Los Alamos National Laboratory
Lines: 56

In article <609@nike.UUCP> lamaster@pioneer.UUCP (Hugh LaMaster) writes:
>Cray users often like to boast that very large main memories such as the
>Cray 2 obviate the need for virtual memory.  J. Giles stated recently that
>Cyber 205 users "turn off" virtual memory.  I work at a site which has a
>Cray X-MP, a Cyber 205, and a Cray-2 (Ames Research Center).  I would like
>to state for the record that there are considerable advantages to having a
>machine have memory mapping virtual memory architecture, even though there
>is no apparent single job performance advantage.  
>
>Picture a very large (almost all of real memory) batch job running on a Cray.
>Suppose that there is some interactive debugging going on (The Cyber 205 VSOS
>operating system and the Cray CTSS operating system are both interactive and
>have very nice interactive debuggers.)  The large batch job will have to be
>written out to disk in entirety on the Cray before the small interactive job
>can be rolled in.  That might take tens of seconds on a Cray 2.  What if only
>a few pages had to be paged out as on the Cyber 205?  The batch job would
>proceed with only a few pages missing, and the total swap time in any case
>would be only a few tenths of a second.  There are many hypothetical examples
>which can be imagined which are confirmed, in my experience, in real life:

You obviously don't have enough Crays!  Now if you had 3 X/MPs (like
we do) you could configure one for interactive use, and the others for
batch use :-).

Seriously: yes there is a trade-off between throughput, interactivity,
and single process speed.  We have to configure our maximum job size
to no more than 3/4 the total memory during the day to allow for
interactive processes.  We also reduce maximum memory residency time
during the day.  We wouldn't have to do this with a virtual memory
system (nor with a total batch operating system).

During the night, when most large scale programs are run, the
operating system is reconfigured to optimize the speen of individual
codes.  On a virtual memory system, this is not possible (at least not
completely - you can't truly rid yourself of the overhead inherent in
the VM interface).  It is the fast turn-around on large night jobs that
is attractive to our users - the ability to also debug interactively
during the day is a useful bonus.

New Crays are coming out now with SSD memory devices (Solid State Disk).
The increased speed of these devices, compared to disk, might make VM
seem to be attractive once more.  Seymore's attitude is (what else):
if it's made of solid state memory - why not make it part of the central
memory of the machine?  The Cray 3 is not currently expected to have an
SSD, but is is expected to fill the entire 32 bit address space with
central memory (that's 32 GB or 4 GW).  Seymore thinks 32 bits is too
small for an address!

This is another problem with virtual memory - central memory is
starting to get cheaper and bigger than the disk memory.  The biggest
disk drive available with a Cray these days is a DD-49 (holds about
151 MW).  One memory image of the 4GW Cray 3 would fill 26 DD-49s to
capacity.  What are you going to operate virtual memory out of?

J. Giles
Los Alamos