Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Posting-Version: version B 2.10.3 4.3bsd-beta 6/6/85; site ptsfa.UUCP
Path: utzoo!watmath!clyde!cbosgd!hplabs!qantel!ptsfa!dsp
From: dsp@ptsfa.UUCP (David St. Pierre)
Newsgroups: net.news.adm,net.news.b
Subject: Re: curious 2.10.3 efficiency situation
Message-ID: <1210@ptsfa.UUCP>
Date: Fri, 14-Mar-86 00:31:24 EST
Article-I.D.: ptsfa.1210
Posted: Fri Mar 14 00:31:24 1986
Date-Received: Sat, 15-Mar-86 19:28:00 EST
References: <5044@glacier.ARPA> <5104@glacier.ARPA> <10428@amdcad.UUCP> <5128@glacier.ARPA> <63@gilbbs.UUCP> <1951@saber.UUCP>
Reply-To: dsp@ptsfa.UUCP (David St. Pierre)
Organization: Pacific Bell, San Francisco
Lines: 107
Xref: watmath net.news.adm:559 net.news.b:1321

In article <1951@saber.UUCP> msc@saber.UUCP (Mark Callow) writes:
>
>There had better be only one instance of uuxqt or you'll have real
>problems.  uuzqt uses the lock file /usr/spool/uucp/LCK.XQT to ensure
>only one uuxqt is running.
>
>UNNNNFortunately uuxqt ignores the lock file if it is more than a
>certain age.  The default age (in 4.2 BSD's uucp at least) is 60
>minutes.  This is far too short if you are receiving batched news,
>especially if you have unbatchnews niced.

For both of these reasons and others, I have taken an extreme approach.
I just checked my (BNU) logfiles and found 1038 executions today. If I
were to let cunbatch grab hours of that time during prime time, I'd
probably be lynched. I can run multiple UUXQTs but don't think it really
helps this problem much.

Here's my "cunbatch":
=====================

cat > news.$$

I just let files drop into my .XQTDIR/.Xqtdir; UUXQT goes off to find more
interesting things to do. I can keep up with incoming traffic without having
queueing problems in UUCP or ageing problems with LCK files. AND I can try to
do some sensible things with incoming news.

Here's something I call "cunbatch.sh" - I'm System V only so it may not work
on your system. Then again it may:
==================================

trap 'rm -f LCK..news; trap 0; exit' 1 2 3 15
cd /usr/spool/uucp/.Xqtdir
if [ -f LCK..news ]
then
	ps -p `cat LCK..news` > /dev/null
	if [ $? -eq 0 ]
	then
		exit 0
	fi
fi

echo $$ > LCK..news

while :
do
	for FILE in `ls -tr news.* 2>/dev/null`
	do
		/usr/lbin/compress -d < $FILE | /usr/lbin/rnews
		rm $FILE
	done
	FILES=`ls -tr news.* 2>/dev/null`
	if [ -z "$FILES" ]
	then
		rm LCK..news
		exit 0
	fi
done


I haven't been too thrilled with system performance or rnews article insertion
speed when there are multiple batch files being processed concurrently. We
run a lot of background daemons on our system which tend to get higher priority
than rnews during the daytime. So I've built a shell LCK-file around the
unbatching process. A second instance of cunbatch.sh finding the first still
running will just go away. If the LCK file is not present or the process is
no longer active, it will start up. A single instance of cunbatch.sh will
process all files present when it started or new ones which arrived after it
started.

I drive cunbatch.sh out of crontab hourly during the evening - it was just
too SLLOOOWWWW during the day. 'Sides, people are supposed to be working
part of the time around here, right?  I can process a day's inputs in 1-2
hours so I don't slow down the sites I feed too much.
Others who don't have those constraints could probably run cunbatch.sh nohup 
right out of cunbatch. You'd still gain the effects of not tying up UUCP 
and being able to single-thread batch file processing.

One additional benefit I derived from the LCK-file processing is that I also
"taught" expire to honor it:
============================

trap 'rm -f $LOCKFILE; trap 0; exit' 1 2 3 15
LOCKFILE=/usr/spool/uucp/.Xqtdir/LCK..news
if [ -f $LOCKFILE ]
then
	ps -p `cat $LOCKFILE` > /dev/null
	if [ $? -eq 0 ]
	then
		echo expire.sh | at now + 30 minutes >>/dev/null
		exit 0
	fi
fi

echo $$ > $LOCKFILE

PATH=/usr/lib/news:$PATH
...shortened for brevity
	expire -e 10
...
> /usr/lib/news/ohistory
rm -f $LOCKFILE

I don't have to worry about one process trying to add to the history file
while another is trying to destroy it.
-- 
David St. Pierre		{ihnp4,dual,qantel,lll-crg}!ptsfa!dsp