Path: utzoo!attcan!uunet!husc6!bloom-beacon!tut.cis.ohio-state.edu!cwjcc!gatech!uflorida!haven!ncifcrf!nlm-mcs!adm!smoke!gwyn
From: gwyn@smoke.ARPA (Doug Gwyn )
Newsgroups: comp.sys.apple
Subject: Re: Streaming Protocols
Message-ID: <8605@smoke.ARPA>
Date: 3 Oct 88 06:05:46 GMT
References: <8809251636.AA04554@crash.cts.com>
Reply-To: gwyn@brl.arpa (Doug Gwyn (VLD/VMB) <gwyn>)
Organization: Ballistic Research Lab (BRL), APG, MD.
Lines: 70

In article <8809251636.AA04554@crash.cts.com> pnet01!pro-sol!pro-newfrontier!pro-freedom!mmcnew@nosc.mil writes:
>You will note in my original message something to the effect that I was not
>talking about Y-MODEM G and if it's the only available 'streaming' protocol
>and it is no faster than plain old XMODEM, then so be it.

I believe this topic started with a question about implementation of
YMODEM-G.  I don't actually know the details of YMODEM-G, but here is
some general information about file transfer protocols:

Assuming data compression is not involved as part of the protocol (it
is not useful if one is transferring already-squeezed files), the
theoretical limit to the data transfer rate is obviously the modem bit
rate.  Assuming that the serial interfaces are compatible with RS-232-C,
each 8-bit data byte will require up to two bits of additional framing
overhead, so 10 bits transferred per 8-bit data byte is the usual
conversion factor.  A synchronous protocol could cut this 25% overhead
to practically zero for large transfers, but let's assume we're not
talking about that.

Additional overhead can come from the way that blocks of data are
packaged for transfer, usually including a block count and checksum or
CRC per each "block"; practical considerations require that blocks not
exceed some reasonable buffer size such as 4K bytes.  Some protocols
negotiate the buffer size and form of checksum to be used before the
actual data transfer starts.  This sort of overhead generally can be
made arbitrarily small by increasing the transfer block size, although
once the blocks are large enough that errors can be expected at a
significant rate, additional increase in block size can actually
decrease throughput due to the need for frequent retransmission of
large chunks of data.  A slight improvement can be made by using an
error-correcting code, but the nature of errors on telephone lines is
such that this can be taken only so far.

Any reliable data transfer protocol will have to have the recipient
acknowledge successful receipt of uncorrupted data and request
retransmission of corrupted data (if error-correcting code is not
being used or is unable to correct the corruption).  In the case of
two-way transmission, such ACKs can be "piggy-backed" inside normal
data packets going in the reverse direction; or, given a full-duplex
connection, ACKs can be sent in the reverse direction while data is
still being transferred in the forward direction.  (An ACK may be a
NAK instead, which calls for retransmission of a corrupted packet.)

XMODEM's ACKs are sent between received data blocks, rather than
while subsequent data is transmitted.  This means that the time per
block needed for the transmitter to receive and check an ACK has to
be added to the overhead time.  For large blocks, this is a small
amount of time and not worth worrying about (my original point).
However, there is one circumstance where this half-duplex mode of
operation (one direction at a time) does present a problem, and
that is with some "9600bps" modems that aren't really 9600bps when
operated as full-duplex (simultaneous transmission in both
directions).  Such modems generally receive "reverse channel"
information at a much lower effective rate, and some have to buffer
data and operate in a genuinely half-duplex manner; they only
APPEAR to be full duplex.  The "turn-around" overhead on these can
easily dominate the transmission time.

There are several ways to set up protocols that make effective use
of such brain-damaged modems.  One obvious one is to continue to
send blocks of data in packets (with packet sequence numbers, CRC
checks, etc.) until the end, at which point an ACK is required, or
until a breakthrough NAK is sent from the recipient to request
retransmission of specified packets.  (Thus each correct packet is
IMPLICITLY ACKed.)  My guess is that the term "streaming protocol"
refers to some scheme like this, as opposed to blocks individually
ACKed like XMODEM (all sizes) uses.

If you get into genuine networking instead of this mickey-mouse
stuff, packet protocol design becomes even more important.