Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!usc!apple!decwrl!decwrl!mogul
From: mogul@jove.pa.dec.com (Jeffrey Mogul)
Newsgroups: comp.protocols.tcp-ip
Subject: Re: When is an ethernet full?
Message-ID: <4597@bacchus.dec.com>
Date: 30 May 90 22:23:00 GMT
References: <1141@cica.cica.indiana.edu> <56724@bbn.BBN.COM>
Sender: news@decwrl.dec.com
Organization: DEC Western Research
Lines: 68


       When is an ethernet full?  ....
       According to our Network General sniffer, 
       we constantly maintain about 10-15 percent utilization or (300 - 1000
       packets per second).   How much more traffic can this network support
       before performance falls off measurably?  Any ideas?  
    
    The best place I know of to start answering this question is Boggs,
    Mogul and Kent's article in Proc. of SIGCOMM '88 pp. 222-233.  The
    gist of that article is that you can drive the Ethernet all the way
    to its rated capacity assuming you are careful in the way you lay out
    your network, and all your systems have good Ethernet hardware.

Thanks for the plug, Craig ... but I think you have misconstrued
our results, at least in trying to apply them to the question at hand.

True, "you can drive the Ethernet all the way to its rated capacity"
(well, at least 95% of the way) if what you are trying to do is to
make full use of the bandwidth.  This is NOT the same thing as saying
that you will have a useful network if the average load is 95%.  In
fact, as I found out last night (while running some TCP benchmarks
on our lab's main Ether) if you use 90%+ of the network between one set
of hosts, other hosts are going to suffer badly.

The reason is queueing delay.  Think of an arbitrary host with a
stream of packets it wants to send.  If the load on the network is
100%, then its output queue will grow forever and the effective delay
will become infinite.  Actually, I think you can show that the
asymptote for infinite delay happens at a load below 100%, for any
finite inter-arrival time for new packets.

What then is the "right" level at which to declare an Ethernet "full"?
That depends.  If you are running a real-time application that can
never accept a delay > 1.2 milliseconds, then you may not be able
to use an Ethernet at all.  If you are only using the net to carry
non-interactive traffic (like electronic mail) then you might get
away with an average load above 90%.  In the usual "NFS+xterm+other
stuff" kind of environment that we run, I've seen 5-second load
averages above 50% without hearing users complain, although I would
probably complain myself if the load stayed this high all the time.
If your average load (calculated over one-second intervals, as is
the usual practice) is only 15%, then you are probably not going
to notice any problems.

The point of our paper is not that you should run your net at 50%
(or 70% or 90%) utilization; we even said ``Don't try this at home.''
The point is that an Ethernet is no worse when carrying high loads
than other 10Mbit/sec multi-access LANs.
    
    In practice of course, many of the systems won't have good Ethernet
    hardware (for example, Jacobson's talk at SIGCOMM '88 indicated he'd
    found an Ethernet chipset that could only go about 6 Mbits/sec).  So
    you need to find some people out there with some good practical experience
    about when some of their systems start breaking down, to figure out when
    your network will die due to poor hardware/software.

In general, even the hosts with the full-speed ethernet interfaces
won't be using them at full speed (because most protocols are
flow-controlled at some level, and the ultimate data sources and
sinks seldom run at 10Mbits/sec.)  If you are worried about worst-case
scenarios, such as somebody like David Boggs or myself running network
benchmarks on your net, then you might want to pay attention to the
capabilities of your host interfaces.  But in most cases, your network
load comes from a composite of many slower sources, and what matters
is how many hosts you have and what fraction of them are going to
be active at once.

-Jeff