Path: utzoo!attcan!uunet!cs.utexas.edu!sun-barr!decwrl!shelby!kolk
From: kolk@shelby.Stanford.EDU (Dan Kolkowitz)
Newsgroups: news.software.nntp
Subject: An interesting hack for nntp feeds
Message-ID: <1081@shelby.Stanford.EDU>
Date: 19 Dec 89 15:31:15 GMT
Reply-To: kolk@shelby.stanford.edu (Dan Kolkowitz)
Organization: Stanford University
Lines: 28



The news configuration at Stanford has a master host feeding several
other hosts, which don't have any other news feeds.  For most of these
hosts we send article batches with rcp, for speed and efficient use of
the network.  We'd like to switch to NNTP both for reasons of security
and more importantly to avoid holding the batches on the master host,
when one of the others is down.

An interesting compromise of the two methods occurred to us: we can
create batches but use NNTP to transmit an entire batch as if it is a
single article.  As long as the message-id is chosen so that there is
no possibility that the batch is rejected, it will be fed to rnews,
and then the batch will be unpacked.  This saves the IHAVE/SENDME
overhead of the protocol and likewise reduces the rnews overhead by
the number of execs of rnews that would have occurred.

The testing that I have done seems to indicate that on a local
Ethernet for an average batch of articles (~200k) it saves about 30% in 
elapsed time over sending it through an
unmodified nntpxmit.  (Our nntpxmit is changed to use a given
message-id instead of one taken from the first article.)  Slower links
should benefit even more from the reduction in protocol overhead.  The
key point is that none of the articles in the batch ever get rejected,
since the server is the sole news source for the recipient machine.
It would not pay if articles were rejected.  I'm just modifying our
shell scripts now to use this method.  If there is interest I'll post
the C code and shell script changes.