Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!rutgers!usc!ucsd!brian
From: brian@ucsd.EDU (Brian Kantor)
Newsgroups: news.software.nntp
Subject: Re: Minutes ... (nntpxmit)
Message-ID: <1816@ucsd.EDU>
Date: 4 Jul 89 12:38:08 GMT
References: <1884@papaya.bbn.com> <121@uakari.primate.wisc.edu> <WISNER.89Jul3112606@anableps.berkeley.edu> <1128@mailrus.cc.umich.edu> <714@stag.math.lsa.umich.edu>
Reply-To: brian@ucsd.edu (Brian Kantor)
Organization: The Avant-Garde of the Now, Ltd.
Lines: 123

I've shipped a series of patches off to Stan Barber that permit you to
limit the number of transfer-mode NNTPd processes running on a system
based on load average.  In particular, nntpd checks two load average
figures and will disallow startup or discontinue transfers if the load
average climbs above the appropriate limits.

The patches also include locking to prevent incoming transfers whilst
'expire' is running;  this is done because there can be no duplicate
article detection when incoming news is spooled into .rnews and with
multiple feeds that's a real waste of resources.

Stan tells me he'll get those changes out when he has time.  Please be
patient.

Below I've included our nntpsend which runs continously; it has load
limiting stuff in it too.  Adjust the load limits and sleeps to suit
your environment.  (UCSD is a poor tired vax/750 and thus not much
faster than an IBM-PC; our primary task is mail so news gets shut off
when the load average impacts mail flow.)  This is started hourly from
crontab in case it exits; shell locking prevents more than one from
running at a time.

Enjoy.
	- Brian
---
#
# What we have here is a csh script for sending netnews to NNTP sites.
#
set stoplim=10
set slowlim=6

set batchdir=/usr/spool/batch libdir=/usr/lib/news
set path=( $libdir /usr/ucb /usr/bin /bin $path )
#
# Go to where the action is
#
cd $batchdir
umask 022
set pname=$0
set pname=$pname:t
#
# Make sure only one of us is running
#
if { shlock -p $$ -f NNTPSEND } then
	echo ${pname}: "[$$]" started `date`
else
	echo ${pname}: "[$$]" already running "[`cat NNTPSEND`]"
	exit 0
endif

while ( 1 )
	set loadavg=( `uptime` )

	# check the 15-minute load average
	if (  $loadavg[$#loadavg]:r > $stoplim  ) then
		echo "load avg too high; quitting"
		echo $loadavg
		rm NNTPSEND
		exit 1
	endif

	set sentsome=0
	echo ${pname}: "[$$]" begin cycle `date`
	foreach host ( \
		rutgers.edu \
		ucbvax.berkeley.edu \
		tut.cis.ohio-state.edu \
		)

		# check the 5-minute load average
		set loadavg=( `uptime` )
		@ n = $#loadavg - 2
		while ( "$loadavg[$n]:r" > $slowlim )
			sleep 600
			# check the 5-minute load average
			set loadavg=( `uptime` )
			@ n = $#loadavg - 2
		end
		
		set lock=NNTP_LOCK.${host} 
		set tmp=${host}.tmp 
		set send=${host}.nntp

		
		if { shlock -p $$ -f ${lock} } then
			if ( -e ${tmp} ) then
				cat ${tmp} >> ${send}
				rm ${tmp}
			endif

			if ( -e ${host} ) then
				if ( -e ${send} ) then
					mv ${host} ${tmp}
				else
					mv ${host} ${send}
				endif
			endif

			if ( -e ${send} ) then
				( echo ${pname}: "[$$]" begin ${host}:${send};\
				time nntpxmit ${host}:${send};\
				rm -f ${lock};\
				echo ${pname}: "[$$]" end ${host}) &
				set sentsome=1
			else
				rm -f ${lock}
			endif
		else
			echo ${pname}: "[$$]" ${host} lock by "[`cat ${lock}`]"
		endif

		# let the load level a bit
		sleep 30
	end

	echo ${pname}: "[$$]" end cycle `date`

	# if there wasn't any news to send this time round, 
	# hibernate for a while
	if (${sentsome} == 0) then
		sleep 300
	endif
end