Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!mailrus!wuarchive!brutus.cs.uiuc.edu!lll-winken!maddog!brooks
From: brooks@maddog.llnl.gov (Eugene Brooks)
Newsgroups: comp.arch
Subject: Re: parallel systems
Message-ID: <36905@lll-winken.LLNL.GOV>
Date: 26 Oct 89 16:17:03 GMT
References: <20764@usc.edu> <36662@lll-winken.LLNL.GOV> <224@dg.dg.com>
Sender: usenet@lll-winken.LLNL.GOV
Reply-To: brooks@maddog.llnl.gov (Eugene Brooks)
Organization: Lawrence Livermore National Laboratory
Lines: 25

In article <224@dg.dg.com> uunet!dg!daver (David Rudolph) writes:
>And what is the overhead for each interconnect level in a "shared
>memory" machine such as the ultracomputer?  Keep in mind that in such a
>machine, every non-local memory access must go through each level and 
>back while the processor waits.
A good figure is a two clock pipeline delay per stage.  A network using
8x8 switch nodes will have 4 stages for a 4096 processor system (why
think small).  Round trip is 16 clocks.  Considering a 40 MHZ clock this
would be a round trip transit time of .4 microseconds.  You would add to
this of course the memory chip cycle time, pipeline delays associated with
getting on and off each end of the each network, and the number of clocks
required to worm hole the message through the available wires.  The memory chip
cycle time and "fixed delays" associated with getting on and off the network
amount to more than the pipeline delay through the network.  A one microsecond
total delay for a remote memory reference (cache miss) is not out of the
question.  A two microsecond delay is trival to achieve. The CPU does not have
to wait for each remote reference to complete before starting another, we
certainly did not stall the processor on every shared memory reference in the
Cerberus multiprocessor simulator, but current KILLER MICROS have a very
limited capability for multiple outstanding memory requests.  Hopefully
we can keep them alive with cache hits for now, and they will get better
about multiple outstanding requests in the future.


brooks@maddog.llnl.gov, brooks@maddog.uucp