Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!ucbvax!GATEWAY.MITRE.ORG!barns
From: barns@GATEWAY.MITRE.ORG
Newsgroups: comp.protocols.tcp-ip
Subject: Re:  high cost of routing
Message-ID: <8908291557.AA03344@arcturus.mitre.org>
Date: 29 Aug 89 15:57:01 GMT
Sender: daemon@ucbvax.BERKELEY.EDU
Organization: The Internet
Lines: 33

It could be many things, and the way to find out for sure is to look
at the packets.  However, here is an example of a possible candidate.
Telnet is an especially difficult protocol to analyze in terms of
theoretical packet behavior.  Neither end is exactly sure when it 
ought to send next, so there are implementation decisions involved.
The decisions found in BSD-flavored code (and many others) tend to
induce a dependency on round-trip time.  If the timing works out
favorably, you will have echoes and acknowledgements and window
updates traveling together.  If it works out less favorably, there
will be extra packets carrying "naked ACKs" and possibly also
window updates.  By going through a router, you increase the RTT by
some amount.  This may increase the chance that the TCP will feel
the need to send an ACK for an incoming segment before the echo or
other output is ready to go back.

Many other things affect packetization, such as Nagle algorithm,
need for retransmission, all the parameters that affect retransmission
(since they indirectly determine what will go in the packets),
context or task switching behavior of the OS, etc.  I've done a few 
limited paper vs. reality studies on what makes packets and find that 
there are so many (nonlinear) factors that even if you know quite a 
lot about the underlying factors, there are likely to be yet more 
factors that you didn't know about.  To summarize, real systems have
a lot of timing-related properties, and the TCP (and perhaps its higher
layer) both contribute to and are affected by them.  These influences 
show up more with irregular flows of small data chunks than with regular 
data flows of big data chunks.

I also feel that the Nagle algorithm is too blunt an instrument to
handle such flows nicely in certain sub-congested regimes, but I
don't think that is what is biting here.

Bill Barns / MITRE-Washington / barns@gateway.mitre.org