Path: utzoo!attcan!uunet!samsung!interlan.InterLan.COM!interlan.interlan.com!solensky
From: solensky@interlan.Interlan.COM (Frank Solensky)
Newsgroups: comp.protocols.tcp-ip
Subject: Re: TCP/IP slowdown and window field of packet
Message-ID: <SOLENSKY.90Jul6132148@interlan.Interlan.COM>
Date: 6 Jul 90 17:21:48 GMT
References: <499@litle.litle.com>
Followup-To: comp.protocols.tcp-ip
Organization: Racal Interlan Inc., Boxborough, MA
Lines: 111
In-reply-to: tom@litle.litle.com's message of 5 Jul 90 18:10:57 GMT

In article <499@litle.litle.com> tom@litle.litle.com (tom hampton) writes:
   We are having a slow-down problem between two TCP/IP programs.

   Both programs do fine for a while, transferring some 50,000 bytes/sec,
   until all of a sudden, they appear to hang and transfer no more than
   1/100th of that.  According to a crude packet monitor we have, the 
   problem seems to occur when the window field of the packet gets set
   down (we don't know by whom) to 0.

Two things come to mind that might explain what you're seeing --

a) a "zero window" problem:
  When the receiver of the data has no space left in its receive window,
the sender stops transmitting, waiting for some indication that it is safe to
resume.  See if the packet trace indicates that:

  1) the receiver sends out another ACK out with a non-zero window size
     shortly after some space opens up in the receive window (does your packet
     monitor display time intervals between the packets?).

  and/or

  2) if the sender sends a "zero-window" probe packet.  The sender should be
     allowing for the possiblity that the above ACK is lost and sends a single
     byte into what it believes to be a full window, expecting that the
     receiver will respond with an ACK with a non-zero receive window.  This
     should occur in about a single retransmit timer interval (see Host
     Requirements for Communications Layers [RFC 1122], section 4.2.2.17 --
     about halfway down page 92).

b) a variant of the "silly window" problem [which I'll refer to as the
   "weird window syndrome"].

   This will show up on your packet monitor as a significant number of "odd"
   sized packets.  Once the receive window has opened up a bit, the window
   size announced back to the sender is some non-intuitve value (eg: not a
   multiple of some power of 2 [or, in the true "silly window" case, some value
   that is very small relative to the receive window size]).  The sender
   transmits enough to fill up the receive window again, and allows the
   transmitting process to unblock until it tries to send at least this amount
   of data again.  This is illustrated below:  "W" is the TCP window size,
   "A" is the ack number, "L" is the length of the packet, "S" is the sequence
   number.

	sender  receiver
        -----   ------

	    <-- W=0, A=5000000
            <-- W=2048, A=5000000    say that a zero-window has just reopened.

 --> send(1024)             S=5006144 is put near end of an 8 KB send window
        L=1024,S=5000000 -->
 --> send(1024)             S=5007168, filling send queue again.
        L=1024,S=5001024 -->

	    <-- W=0, A=5002048
            <-- W=1662 (??), A=5002048

        L=1024,S=5002048 -->
        L= 638,S=5003072 -->   the first part of a queued 1 KB buffer is sent

            <-- W=3092, A=5003710

                     User's send window is now opened only to 1662 bytes.

        L= 386,S=5003710 -->   the end of 2nd buffer is sent but not yet acked.

 --> send(1024)      New requests from host application allow 1024-byte and
 --> send(1024)      638-byte buffer to be pulled into send queue.  The tail
                     of the buffer is pulled down after some space opens up
                     again in the send window.

           . . .
        L=1024 -->
        L=1024,S=5006144 -->  first send() from above
        L= 638,S=5007168 -->  second send()
        L= 386,S=5007806 -->  the remainder of second, once sender was
				unblocked.


     Soon afterwards, the system falls into a self-perpetuating cycle of
filling its send window with a weird-sized packet, transmitting and getting
an ACK back that might not correspond to the boundary of yet another buffer.
The sender's performace suffers as a result of trying to keep packets in synch
with both the announced window sizes and the implied boundaries between each
send() request in the transmitting application program (since the boundaries
of each entry on the send queue closely resemble the send requests of the
initiating process).

     I've seen packet traces a few months ago where this occurs with one of
the vendors you mentioned, but don't recall offhand what rev of software was 
being used at the time.  The lower bound of the performance degredation was
approximately where "silly window" conditions were matched -- about 35% of
the receive window.  I had brought their announced window size to their
attention at the time, so maybe the problem will be fixed Real Soon Now
(though, obviously, I'm not in a position to announce what their plans are).

     The way we worked around this is that when some but not all of
the data in the send queue is acknowledged, reduce the bytes-acked count
in tcp's input so that a queued buffer isn't "partially" acknowledged.  This
forces the send routine to hold off pulling weird-sized packets into memory
in the first place:  here, the second 638-byte packet shouldn't occur, since
the space in the send window won't be released until the rest of the data in
the first "fragmented" packet packet is acknowledged, thereby preventing the
send() request that had created the perpetuating fragment from starting.

     Hope this helps..

--
					-- Frank Solensky / Racal InterLan
Red Sox magic number: 80