Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sdd.hp.com!elroy.jpl.nasa.gov!usc!rpi!batcomputer!cornell!ken From: ken@CS.Cornell.EDU (Ken Birman) Newsgroups: comp.sys.isis Subject: Re: lazy replication Message-ID: <1991Mar26.191407.13438@cs.cornell.edu> Date: 26 Mar 91 19:14:07 GMT References: <53124@cornell.UUCP> <1991Mar26.033004.25436@mintaka.lcs.mit.edu> Sender: news@cs.cornell.edu (USENET news user) Distribution: comp Organization: Cornell Univ. CS Dept, Ithaca NY 14853 Lines: 57 Nntp-Posting-Host: gvax.cs.cornell.edu In article <1991Mar26.033004.25436@mintaka.lcs.mit.edu> liuba@proton.lcs.mit.edu (Liuba Shrira) writes: > >... First, the Lazy Replication paper that appeared in PODC 90 does not >contain any performance data.... Actually, my comments were based on Sec. 3.6.1 of the PODC paper, which did give some numbers. >... the highly engineered ISIS system! Thanks... but, it is funny to read this even as I type in a revision to our TOCS paper explaining why ISIS multicast is so much slower than Ameoba or X-kernel multicasts! Both systems are losing heavily by running over UNIX -- my guess is that Argus is only a minor problem. The situation we see in ISIS -- I wonder if you run into this too -- is that UNIX tends to throw away a lot of UDP messages if you present it with lots of data at once. For example, we have been looking at how ISIS performance varies as you stream data to larger and larger process groups. What we find is that the dominent effect occurs when UNIX` suddenly decides that there isn't enough mbuf memory left, and starts to discard both outgoing messages, from the sender, and also incoming ones (in this kind of test, mostly acknowledgement packets). Unfortunately, UNIX usually doesn't indicate that it is doing this, and it never indicates when it discards an incoming packet, except to keep a count in some internal, undocumented, kernel data structure. Packet loss is a performance disaster, and to make matters worse, there seems to be no way to figure out that UNIX is doing this. You can tell that it is happening -- running netstat -m, for example -- but at runtime there is no UNIX system interface to indicate that UNIX is getting overfilled with data. So, you give it the data, it thanks you and discards it, and then you sit and wait to see if it went anywhere. Needless to say, this causes a lot of multi-second delays, waiting for acks that don't make it, retransmitting packets, etc. Our course, there are flow control heuristics that help, a little, but my belief is that when we measure ISIS or LR or ARGUS over UNIX, we are so far from what the hardware can do and running over such a complex and quirky black box, that the situation is really pretty hopeless. For ISIS, the plan is to rebuild our system over the X-kernel inside Mach, or inside Chorus (a comparable environment), as close to the hardware as possible. This should give us direct access to information like memory usage in the kernel and to hardware multicast, and will make a huge difference to our system performance. Is anyone aware of successful multicast mechanisms layered over UNIX? I would be very interested to hear about other experiences... -- Ken -- Kenneth P. Birman E-mail: ken@cs.cornell.edu 4105 Upson Hall, Dept. of Computer Science TEL: 607 255-9199 (office) Cornell University Ithaca, NY 14853 (USA) FAX: 607 255-4428