Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!csd4.milw.wisc.edu!uxc!uxc.cso.uiuc.edu!mcdurb!aglew
From: aglew@mcdurb.Urbana.Gould.COM
Newsgroups: comp.arch
Subject: Re: Time Sync in MP System
Message-ID: <28200330@mcdurb>
Date: 4 Jun 89 19:23:00 GMT
References: <2280007@hpsal2.HP.COM>
Lines: 62
Nf-ID: #R:hpsal2.HP.COM:2280007:mcdurb:28200330:000:3014
Nf-From: mcdurb.Urbana.Gould.COM!aglew    Jun  4 14:23:00 1989


>How can I initialize and maintain time synchronization in a
>multiprocessor system?  To put this in context, I am seeking an
>approach that can be used by the IEEE CSR Standard, which will
>be included by reference in Futurebus+ and SCI.  
>
>We have a 64 bit system clock counter register in each node in
>the system, which must be initialized  --  and resynchronized due
>to local oscillator drift.  The clock counts in nanoseconds, but
>nodes can increment by larger quantities if they lack a GHz
>oscillator   :<)

Do not resynchronize the counter!  That sort of resynchronization
leads to "time warps" which dramatically reduce the usefulness
of the clock counter for high resolution time measurements,
performance evaluation, etc.

Instead, implement both the counter register and a correction register.
The correction register has to be added to the counter to get a
"time of day" measurement - but, the raw time in whatever rate the
local oscillator is ticking at can always be read.
    For small interval measurements the drift of the oscillator is 
probably less than the "warp" that you would apply to resynch the
timers.

Provide ways of reading both the raw time and the correction value
together. This does not have to be an atomic operation, but it should
be fairly precise.
    Note that the correction register changes much less often than
the local clock counter, so it is not unreasonable to define a
READ_CLOCK_AND_CORRECTION operation as two separate reads.
    Note also that providing both a raw and correction counter
avoids a race for incrementing the counter - the raw counter
is incremented only by the oscillator, the correction only by the
time synch interface.

(Actually, you would probably want a raw clock, current correction, 
and a deviation, with the deviation slowly "warping" its way into the
correction register in such a manner that raw+correction is a 
monotonic time of day clock suitable for timestamping. raw is useful
for interval measurements, and raw+deviation is the truest estimate
of the time of day).


I may sound a bit cock-eyed here, but I am deadly serious!
Please talk to someone who has actually tried to do very-high-res
measurements before defining a clock standard - myself, 
Agrawala at Maryland or Thareja at A&T. Maybe somebody in the 
POSIX Real Time group (although I don't think that there are too 
many performance people there anymore). This is exactly the sort
of thing that we were talking about at the UNIX Performance
Evaluation Birds-of-a-Feather at SIGMETRICS '89 (the BOF was a 
bit of a flop, but we did bring some ideas forward).


Andy "Krazy" Glew   aglew@urbana.mcd.mot.com   uunet!uiucdcs!mcdurb!aglew
   Motorola Microcomputer Division, Champaign-Urbana Design Center
	   1101 E. University, Urbana, Illinois 61801, USA.
   
My opinions are my own, and are not the opinions of my employer, or
any other organisation. I indicate my company only so that the reader
may account for any possible bias I may have towards our products.