Path: utzoo!utstat!helios.physics.utoronto.ca!jarvis.csri.toronto.edu!mailrus!iuvax!maytag!looking!brad
From: brad@looking.on.ca (Brad Templeton)
Newsgroups: news.software.b
Subject: Re: Dynamic "smart" expiration?
Message-ID: <69448@looking.on.ca>
Date: 30 Dec 89 18:10:48 GMT
References: <1989Dec27.033817.9953@smsc.sony.com> <1989Dec28.063932.13720@robohack.UUCP> <68634@looking.on.ca> <1989Dec29.213539.2801@utzoo.uucp> <6118@yunexus.UUCP>
Organization: Looking Glass Software Ltd.
Lines: 38
Class: discussion

Perhaps the idea is to break up expire neatly into two parts.  One prepares
the list of articles to go according to whatever criterion, and the other
removes them, updates the database and active file etc.


The removing part could be part of inews, or an independent program.

I haven't done anything, but it seemed to me that if you want a really fancy
expire, something like a newsclip program might make an interesting front
end.  You could keep or expire or weight articles based on anything -- from
what group they're in, to who posted them, to what thread they're in to
whether they contain patterns.

Of course, it needn't be newsclip, it could be any scanning program, from
those that just read the history file to anything else you want to code.

The bad part is that this of course requires reading every article, which was
the slow thing about B news expire.   Fortunately, once you calculate the
score for an article, the only thing that affects it is the passage of time,
so you could arrange to keep a history like file with calculated scores,
the time they were calculated, and the multiplier to be used when adding
the time since to get the final score.   In this case, you only have to
scan new articles and add to the file.


This is not enough, however.  The program that sorts and decides who is
first to go needs some smarts beyond this.  The simplest thing to do is
just keep the N blocks of articles with the highest (lowest?) scores.

But if you want to get fancy, you might want to instead assign Y blocks
per group.  ie. news.software.b always keeps 300K, talk.bizarre always
keeps 150K, etc.   You might also want to keep a fixed number of articles
in a group. -- ie. keep 10 articles in groups that nobody currently reads,
or keep a minimum of 5 articles in any group, even if it's super-low volume.

All in all, a messy problem...
-- 
Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473