Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Posting-Version: version B 2.10.3 4.3bsd-beta 6/6/85; site hpda.UUCP
Path: utzoo!watmath!clyde!burl!ulysses!gamma!epsilon!zeta!sabre!petrus!bellcore!decvax!decwrl!pyramid!pesnta!hplabs!hpda!bd
From: bd@hpda.UUCP (Bob Desinger)
Newsgroups: net.news.adm
Subject: Setting up a compressed+batched newsflow
Message-ID: <1282@hpda.UUCP>
Date: Fri, 10-Jan-86 20:28:05 EST
Article-I.D.: hpda.1282
Posted: Fri Jan 10 20:28:05 1986
Date-Received: Tue, 14-Jan-86 04:23:56 EST
Reply-To: bd@hpda.UUCP (Bob Desinger)
Organization: Hewlett-Packard Co., Cupertino, CA
Lines: 496
Keywords: compress, batch, 2.10.3, expire

I recently installed 2.10.3 news on hpda; Greg Fowler of hplabs showed
me how to set up a compressed and batched newsfeed to his system and
back.  The black magic of that afternoon was impressive, and I wanted
to share the knowledge.  So I wrote down these (hopefully easy to
follow) steps detailing how to transmit and receive compressed news
batches.

The people who tested these instructions were also using 2.10.3beta, so
this cookbook applies to that version.  I believe it should also work
on 2.10.2, but in honesty I haven't brought up 2.10.2 news.  (Perhaps
some 2.10.2 wizard will post mods to this document; hope springs
eternal.) 

Not long after setting up news, I decided to expire some of the
newsgroups sooner than the default in order to save disk space.  This
10-minute task took an afternoon because of the gotchas.  So one of the
files below explains how to expire some groups sooner.   You can tell
which groups are the biggest hogs with the commands:

	cd /usr/spool/news	# or wherever you keep your articles
	du | sort -rn		# show biggest directories first

Hopefully this will save netnews administrators some time and grief.
Rick Adams has this shar, so perhaps these documents (or forms of them)
will appear in the 2.10.3 documentation.

Bob Desinger		ihnp4!ucbvax!hpda!bd		hpda!bd@BERKELEY.EDU


# This is a shell archive.  Remove anything before this line,
# then unpack it by saving it in a file and typing "sh file".
#
# Wrapped by hpda!bd on Fri Jan 10 16:44:20 PST 1986
# Contents:  README batch.lpr batch.ms expire-sooner
 
echo x - README
sed 's/^@//' > "README" <<'@//E*O*F README//'
Three files are included below:

1.  batch.lpr	The post-nroff cookbook, suitable for feeding to lpr (or
		lp, depending on your Unix version) directly.

2.  batch.ms	The pre-nroff cookbook; process via `nroff -ms batch.ms'.

3.  expire-sooner	A plain file describing how to expire some of the
		less-useful newsgroups sooner than normal.
@//E*O*F README//
chmod u=rw,g=r,o=r README
 
echo x - batch.lpr
sed 's/^@//' > "batch.lpr" <<'@//E*O*F batch.lpr//'






_0._1.  _C_o_o_k_b_o_o_k:  _S_e_t_t_i_n_g _u_p _a _C_o_m_p_r_e_s_s_e_d, _B_a_t_c_h_e_d _N_e_w_s_f_e_e_d

     First, BATCH must have been #define'd  when  you  built
the  news  system.  To check, look in the file defs.h in the
news source directory.  BATCH should be defined there  as  a
program  name (by default, "unbatch").  If it's undefined or
commented out, define  it,  re-make  the  news  system,  and
install the new software.

     You'll also need a working _c_o_m_p_r_e_s_s program.   Use  the
one  shipped  with this news distribution, which is based on
version 3.0; or  use  compress  4.0,  which  was  posted  to
mod.sources  a  while  back.   Your news neighbors should be
running a compatible version of compress.  Versions 3.0  and
4.0  are compatible with each other, but both are incompati-
ble with versions 2.0 and before.

     Update your _s_y_s file.  First, add the  F  flag  to  the
other   news   system's   line.    For   instance,  if  your
compressed-and-batched news feed is named frobozz,  and  its
_s_y_s file entry looks like:

        frobozz:net,mod,na,usa,ca,to.frobozz::

then add the F flag as the third (colon-separated) field:

        frobozz:net,mod,na,usa,ca,to.frobozz:F:

Now the pathnames of articles to be sent will be stashed  in
a  file.   This file is named in the fourth field of the _s_y_s
entry;  add  it   now.    Use   an   entry   of   the   form
_B_A_T_C_H_D_I_R/_s_y_s_t_e_m,  where _B_A_T_C_H_D_I_R is usually /usr/spool/batch
(the actual value is defined in the news Makefile), and _s_y_s_-
_t_e_m  is  the name of the remote system, in this example fro-
bozz.  A name of that  form  is  necessary:  the  _c_s_e_n_d_b_a_t_c_h
script,  which  compresses and sends the batched news, looks
for a file name of this form to decide if there's  news  for
the remote system.

     Your completed _s_y_s  file  line  should  look  something
like:

        frobozz:net,mod,na,usa,ca,to.frobozz:F:/usr/spool/batch/frobozz


     In /usr/lib/crontab, find or create at least  two  news
lines:  one  that runs nightly, and one that runs every hour
or so.  The nightly-run script should run _e_x_p_i_r_e,  trim  log
files,  and  perhaps compile weekly statistics that you post
to a local-area newsgroup one day a  week.   The  hourly-run
script  should  complete  the  transmitting task with a line
like:

        csendbatch frobozz



                      January 3, 1986





                           - 2 -


Make sure the script knows how to get to  the  directory  in
which  _c_s_e_n_d_b_a_t_c_h  lives.  You can either mention the direc-
tory in the script's PATH-setting line, or  replace  "csend-
batch"  with  its full pathname.  _C_s_e_n_d_b_a_t_c_h reads the files
mentioned   in   /usr/spool/batch/frobozz,   batches   them,
compresses  them,  sends  them  to  the  remote  system, and
arranges for remote processing.

     This remote processing is directed by another  file  in
_B_A_T_C_H_D_I_R.    Make   a   file   with   a  name  of  the  form
_B_A_T_C_H_D_I_R/_s_y_s_t_e_m.cmd        (for        this         example,
/usr/spool/batch/frobozz.cmd).   Put a line in it specifying
the command that the remote system should execute to  unpack
the  news  batches  that your system will send.  Sample con-
tents for frobozz.cmd are:

        uux - -r -z -n -gd frobozz!cunbatch


     Now your system will transmit compressed batches.   The
receiving  side of the business is handled largely by a pro-
gram called _c_u_n_b_a_t_c_h.

     Make sure there is an executable file  called  _c_u_n_b_a_t_c_h
in  the  BINDIR directory (check the Makefile for its actual
location).  It must be reachable  by  uucp  or  by  whatever
transport  you'll  use  to  transfer  the  netnews.   If you
defined BINDIR as /usr/bin,  you  should  have  no  problems
because uuxqt can already get there.  If you defined it as a
different directory, you may have to teach uuxqt to look  in
that  directory;  accomplishing  this  varies from system to
system.  On 4.2BSD, add the directory to the "PATH=" line of
your uucp L.cmds file.  On System V, on the _c_u_n_b_a_t_c_h line of
your L.cmds  file,  add  a  comma  followed  by  the  remote
system's    name   on   that   line.    If   yours   is   in
/usr/bin/news/cunbatch, your L.cmds file will look like:

        [For 4.2BSD]
        PATH=/bin:/usr/bin:/usr/bin/news
        cunbatch

        [For System V]
        /usr/bin/news/cunbatch,frobozz

Other systems have  a  similar  file  in  the  /usr/lib/uucp
directory  by which you can specify added programs and paths
different from the defaults.   HP-UX,  for  example,  has  a
/usr/lib/uucp/COMMANDS  file which expands uuxqt's horizons.
In more restrictive cases, paths are  compiled  into  uuxqt.
If  you  can't  modify  any uucp files, just put _c_u_n_b_a_t_c_h in
/usr/bin.

     You must also have a _c_u_n_b_a_t_c_h in _L_I_B_D_I_R (wherever  your
Makefile  defines  it), because _r_n_e_w_s will eventually try to



                      January 3, 1986





                           - 3 -


exec that copy.

     Tell the person at the other end of  your  newsfeed  to
use  _c_s_e_n_d_b_a_t_c_h  and  a  uux command of _c_u_n_b_a_t_c_h to send you
news.  Once that's in place, watch  your  uucp  LOGFILE  and
your  news _l_o_g and _e_r_r_l_o_g files to ensure that news is being
correctly received and unpacked on your system.


















































                      January 3, 1986


@//E*O*F batch.lpr//
chmod u=rw,g=r,o=r batch.lpr
 
echo x - batch.ms
sed 's/^@//' > "batch.ms" <<'@//E*O*F batch.ms//'
@.if t .po 0.5i
@.if t .ll 7.5i
@.NH 2
Cookbook:  Setting up a Compressed, Batched Newsfeed
@.PP
First, BATCH must have been #define'd when you built the news system.
To check, look in the file
defs.h
in the news source directory.
BATCH should be defined there as a program name (by default,
\*Qunbatch\*U).
If it's undefined or commented out,
define it, re-make the news system, and install the new software.
@.PP
You'll also need a working
@.B compress
program.
Use the one shipped with this news distribution,
which is based on version 3.0;
or use compress 4.0, which was posted to mod.sources a while back.
Your news neighbors should be running a compatible version of compress.
Versions 3.0 and 4.0 are compatible with each other,
but both are incompatible with versions 2.0 and before.
@.PP
Update your
@.B sys
file.
First, add the F flag to the other news system's line.
For instance, if your compressed-and-batched news feed is named frobozz,
and its
@.B sys
file entry looks like:
@.DS
frobozz:net,mod,na,usa,ca,to.frobozz::
@.DE
then add the F flag as the third (colon-separated) field:
@.DS
frobozz:net,mod,na,usa,ca,to.frobozz:F:
@.DE
Now the pathnames of articles to be sent will be stashed in a file.
This file is named in the fourth field of the
@.B sys
entry; add it now.
Use an entry of the form
@.I BATCHDIR/system ,
where
@.I BATCHDIR
is usually
/usr/spool/batch
(the actual value is defined in the news Makefile), and
@.I system
is the name of the remote system,
in this example
frobozz.
A name of that form is necessary: the
@.B csendbatch
script,
which compresses and sends the batched news,
looks for a file name of this form
to decide if there's news for the remote system.
@.PP
Your completed
@.B sys
file line should look something like:
@.DS
frobozz:net,mod,na,usa,ca,to.frobozz:F:/usr/spool/batch/frobozz
@.DE
@.PP
In /usr/lib/crontab, find or create at least two news lines:
one that runs nightly, and one that runs every hour or so.
The nightly-run script should run
@.B expire ,
trim log files,
and perhaps compile weekly statistics
that you post to a local-area newsgroup one day a week.
The hourly-run script should complete the transmitting task
with a line like:
@.DS
csendbatch frobozz
@.DE
Make sure the script knows how to get to the directory in which
@.B csendbatch
lives.
You can either mention the directory in the script's
PATH-setting line, or replace
\*Qcsendbatch\*U
with its full pathname.
@.B Csendbatch
reads the files mentioned in /usr/spool/batch/frobozz,
batches them, compresses them, sends them to the remote system,
and arranges for remote processing.
@.PP
This remote processing is directed by another file in
@.I BATCHDIR .
Make a file with a name of the form
@.I BATCHDIR/system .cmd
(for this example,
/usr/spool/batch/frobozz.cmd).
Put a line in it specifying the command that the remote system
should execute to unpack the news batches that your system will send.
Sample contents for
frobozz.cmd
are:
@.DS
uux - -r -z -n -gd frobozz!cunbatch
@.DE
@.PP
Now your system will transmit compressed batches.
The receiving side of the business is handled largely by a program called
@.B cunbatch .
@.PP
Make sure there is an executable file called
@.B cunbatch
in the
BINDIR
directory (check the Makefile for its actual location).
It must be reachable by uucp
or by whatever transport you'll use to transfer the netnews.
If you defined
BINDIR
as
/usr/bin,
you should have no problems because uuxqt can already get there.
If you defined it as a different directory,
you may have to teach uuxqt to look in that directory;
accomplishing this varies from system to system.
On 4.2BSD, add the directory to the
\*QPATH=\*U
line of your uucp
L.cmds
file.
On System V, on the
@.B cunbatch
line of your
L.cmds
file, add a comma followed by the remote system's name on that line.
If yours is in
/usr/bin/news/cunbatch,
your
L.cmds
file will look like:
@.DS
[For 4.2BSD]
PATH=/bin:/usr/bin:/usr/bin/news
cunbatch
@.sp
[For System V]
/usr/bin/news/cunbatch,frobozz
@.DE
Other systems have a similar file in the
/usr/lib/uucp
directory by which you can specify added programs
and paths different from the defaults.
HP-UX, for example, has a
/usr/lib/uucp/COMMANDS
file which expands uuxqt's horizons.
In more restrictive cases, paths are compiled into uuxqt.
If you can't modify any uucp files,
just put
@.B cunbatch
in
/usr/bin.
@.PP
You must also have a
@.B cunbatch
in
@.I LIBDIR
(wherever your Makefile defines it),
because
@.B rnews
will eventually try to exec that copy.
@.PP
Tell the person at the other end of your newsfeed
to use
@.B csendbatch
and a uux command of
@.B cunbatch
to send you news.
Once that's in place, watch your uucp LOGFILE and your news
@.B log
and
@.B errlog
files to ensure that news is being correctly received and unpacked
on your system.
@//E*O*F batch.ms//
chmod u=rw,g=r,o=r batch.ms
 
echo x - expire-sooner
sed 's/^@//' > "expire-sooner" <<'@//E*O*F expire-sooner//'
Some administrators prefer to squeeze every ounce of disk space
possible from their disk drives.  Because the netnews takes up so much
space, non-technical but high-volume groups are sometimes expired more
rapidly than technical newsgroups.  You can do this by forcing `expire'
to delete articles in certain newsgroups after N days, where N is
typically equal to 7.

As an example, the news expiration lines in the nightly news-cleanup
job run by cron are:

	expire -v -e7 -n net.flame,net.jokes,net.politics,net.rec,
	expire -v

The first line expires articles in the specified newsgroups after 7
days.  The second line expires articles in all newsgroups after the
default period, which is 14 days.

Important things to note about running `expire' on a selected list of
newsgroups are:

1.  After the "-n" flag, commas (not spaces) separate the newsgroups in
    the list.

2.  A comma after the last newsgroup in the list is absolutely necessary.

3.  Newsgroups on the `expire' argument line specify a name-pattern for
    newsgroups, and are not necessarily actual newsgroup names
    themselves.  Using the example on the above line, the `net.jokes'
    pattern causes articles in all groups beginning with the title
    "net.jokes" to be expired faster.  Thus articles in net.jokes,
    net.jokes.d, and other newsgroups beginning with "net.jokes" will
    expire sooner.  Likewise with the `net.rec' specifier: net.rec,
    net.rec.bird, net.rec.nude, and other "net.rec*" newsgroups will
    expire sooner.
@//E*O*F expire-sooner//
chmod u=rw,g=r,o=r expire-sooner
 
echo Inspecting for damage in transit...
temp=/tmp/shar$$; dtemp=/tmp/.shar$$
trap "rm -f $temp $dtemp; exit" 0 1 2 3 15
cat > $temp <<\!!!
       9      50     338 README
     198     756    5714 batch.lpr
     184     796    4762 batch.ms
      34     223    1515 expire-sooner
     425    1825   12329 total
!!!
wc  README batch.lpr batch.ms expire-sooner | sed 's=[^ ]*/==' | diff -b $temp - >$dtemp
if [ -s $dtemp ]
then echo "Ouch [diff of wc output]:" ; cat $dtemp
else echo "No problems found."
fi
exit 0