Path: utzoo!mnetor!geac!torsqnt!news-server.csri.toronto.edu!math.lsa.umich.edu!zaphod.mps.ohio-state.edu!usc!cs.utexas.edu!uunet!sceard!mrm
From: mrm@sceard.Sceard.COM (M.R.Murphy)
Newsgroups: news.software.b
Subject: Re: Is there a way to expire to a specified free disk space? (LONG)
Summary: yes :-)
Keywords: expire, space
Message-ID: <1990Nov8.004259.10413@sceard.Sceard.COM>
Date: 8 Nov 90 00:42:59 GMT
References: <1990Nov6.181747.21191@bhpcpd.kembla.oz.au>
Reply-To: mrm@Sceard.COM (M.R.Murphy)
Organization: Sceard Systems, Inc.  San Marcos, CA  92069
Lines: 173

In article <1990Nov6.181747.21191@bhpcpd.kembla.oz.au> bernd@bhpcpd.kembla.oz.au (Bernd Wechner) writes:
>I have often wondered if there was a nice easy way to expire news until
>a specified quantity of disk space if freed. This would help to maintain
>free disk space which is currently consumed when news breaks through a
>bottleneck in the network. Under this circumstance, an expire which
>keeps news for a specified period of time will let the disk overfill. 
>
>In essence it would be nice to say:
>
>"expire as usual, but keep going if less than X MB of free space exists"
>
>Has this been done? Have I not RTFM closely enough?
>[...]

I'm sorry this is so long, it started out to be a short reply.

I've been running a non-traditional expire under C News for a while. I figured
that expiration should be a multi-part thing:

  1) keep history of stuff that has come by so that duplicates are not saved
     or propagated (it is important to keep history of articles that are no
     longer stored in the news spool),
  2) keep enough space in the news spool so that incoming articles like the
     important things in news.groups can be accepted,
  3) get rid of old articles so that new articles like the important things
     in alt.flame can be accepted,

1) above is the responsiblility of expire. 2) and 3) above are the responsiblity
of a "trasher".

The expire that comes with C News doesn't work too well in small model :-)
However, getdate, spacefor, awk, and find work pretty slick, so...
For that reason and for understandability, I replaced expire with a script
that goes through the history file, and if the date and time is older than some
desired date, don't write the line to a new history file. That is, the script
sifts out the old stuff from history. It doesn't bother about deleting any of
the files in a history line, it just worries about history. When the new file
is complete, then new dbm (or in my case, dbz) files are created. Save the old,
copy the new to the old, delete the saved old if you are short on space, and
history is ok. The important part of the expire script is:
-----
cd $NEWSCTL
[ $makehistory = "yes" ] && {
rm -f history.n history.n.pag history.n.dir
now=`getdate now`
ago=`awk "/^\/expired\// {print ($now-(86400*\$(3)))} {next}" explist`
awk "{split(\$2,dates,\"~\");if(dates[1]>$ago)print \$0}" history >history.n
mkdbm history.n
[ -s history.n ] &&
mv history history.o &&    # install new ASCII history file
mv history.n history &&
rm -f history.pag &&       # and related dbm files
rm -f history.dir &&
mv history.n.pag history.pag &&
mv history.n.dir history.dir
-----
This makes the creation of a new history file pretty simple. That's all that
this expire does. Note that it grabs the history time from explist, which is
otherwise unused.

The trasher gets rid of old articles. If there is enough space on the news
spool, there is no need to trash, otherwise

  1) trash the worthless stuff first,
  2) trash the stuff that only deserves keeping for a while,
  3) trash from history starting with the oldest first. Since history
     is automagically in order from oldest first, this is pretty easy.
  4) trash everything based on age (including in.coming and out.going,
     why play favorites :-)

To trash is to remove news articles until a high limit is reached.
The high limit should be higher than the low limit :-) This prevents
unnecssary thrashing.

The important part of the trasher looks like this:
-----
TRASH=${NEWSCTL}/trashlist
# make HILIM - LOLIM about a day or so of news in blocks
LOLIM=25000
HILIM=50000
# if below low limit, trash files until high limit reached
LC=`spacefor 512 articles`
if [ $LC -lt $LOLIM ] ; then
  cat $TRASH | while read GRP FLGS
  do
    LC=`spacefor 512 articles`
    if [ $LC -lt $HILIM ] ; then
      find $GRP -type f $FLGS -print 2>/dev/null|xargs rm -f
    else
      break
    fi
  done
  LC=`spacefor 512 articles`
fi

# if still below low limit, remove oldest files from history ( inefficient )
if [ $LC -lt $LOLIM ] ; then
  sed -e 's/.*  //' -e '/^$/d' -e '/control/d' -e 's/\./\//g' $NEWSCTL/history |
  while read art arts
  do
    if [ $LC -lt $HILIM ] ; then
      [ -f "./$art" ] && {
        rm -f $art $arts
        LC=`spacefor 512 articles`
        }
    else
      break
    fi
  done
else
  break
fi
-----

Our $NEWSCTL/trashlist looks like (order counts):
-----
in.coming -mtime +5
out.going -mtime +5
psu
alt.sex
junk -mtime +1
control -mtime +3
talk -mtime +1
soc -mtime +1
alt.flame -mtime +1
alt -mtime +4
trial -mtime +5
news.groups -mtime +2
news.lists -mtime +3
comp.os.vms -mtime +2
comp.mail.maps -mtime +2
comp.org.fidonet
comp.binaries.amiga
comp.binaries.apple2
comp.binaries.atari
comp.binaries.hypercard
comp.binaries.mac
comp.binaries.os2
comp.binaries -mtime +1
comp.sys.amiga
comp.sys.apple2
comp.sys.atari
to -mtime +10
sci -mtime +2
rec -mtime +2
ca -mtime +14
comp -mtime +5
gnu -mtime +5
junk
la -mtime +5
misc -mtime +5
news -mtime +5
rec -mtime +5
sci -mtime +5
sdnet -mtime +5
soc -mtime +5
talk -mtime +5
to -mtime +5
unix-pc -mtime +5
-----

This is our trashlist.  If your favorite group is slighted, please accept my
apology. If you leave out an heirarchy, it doesn't get expired. It can grow big.
If you leave out an mtime, then everything in the heirarchy goes. You can put
in an heirarchy more than once with decreasing times. Lots of ways to do what
you want to do and to fine tune. To say that this isn't as well checked out as
the rest of C News is a mild understatement, be careful, the risk is yours.

If anyone feels like putting in fancier flags in trashlist, say to provide
archiving before removal, it should be pretty easy. What have I done wrong?
Let me know.
-- 
Mike Murphy  mrm@Sceard.COM  ucsd!sceard!mrm  +1 619 598 5874


Brought to you by Super Global Mega Corp .com