Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!uunet!looking!brad
From: brad@looking.on.ca (Brad Templeton)
Newsgroups: news.software.b
Subject: Re: Steady state news.
Message-ID: <1990Dec12.213956.6544@looking.on.ca>
Date: 12 Dec 90 21:39:56 GMT
References: <Faivp#r3@cs.psu.edu>
Organization: Looking Glass Software Ltd.
Lines: 43

Yes, I have long thought that expire-as-it-arrives is the best solution.

You don't have to be stupid about it either.

For example, you could have a process run nightly (or less than that, as
need be) which prepares a list of articles on the system, sorted by their
"value"

The value is up to you, but it would no doubt be a factor of their age,
newsgroup, author/site (ie. keep local articles longer) modified by things
like explicit expiry, etc.   You can, of course, get age, explicit expiry,
newsgroup and posting site from the current C news history file.

You need one other thing, namely size.  (My space based expire use to
factor size in as well, reducing the value of big files, so that they
went slightly earlier -- theory being it was better to toss one 40K article
than 20 2K articles.   This might not apply in source groups.)

Anyway, you sort the articles based on their value, and you thus produce
a list, from the least valuable upwards, with the message-id and size in
disk blocks.

(You might add more to make it go faster on subsequent nights, since you
could, in theory, calculate value on new articles, figuring the value
of known articles from their old value and the elapsed time.)

On the other hand, you only need to store the least valuable N megabytes of
articles.

When your news database program (relaynews) comes along, it counts the
disk space it uses.   As it uses space, it goes through the expire file
and frees up enough files to get back that space -- keeping track of
extra stuff freed in some sequence file.   (It needs to store a seek
address into the value file too.)

And thus news takes exactly a fixed amount of space, but with some groups
going faster than others, etc. etc.

Since the purpose of expire is to keep news to a limited amount of disk
space, and not a limited number of days, this seems to me the ideal way
to do an expire.
-- 
Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473