Path: utzoo!utstat!helios.physics.utoronto.ca!jarvis.csri.toronto.edu!cs.utexas.edu!usc!zaphod.mps.ohio-state.edu!mips!smsc.sony.com!dce
From: dce@smsc.sony.com (David Elliott)
Newsgroups: news.software.b
Subject: Dynamic "smart" expiration?
Message-ID: <1989Dec27.033817.9953@smsc.sony.com>
Date: 27 Dec 89 03:38:17 GMT
Reply-To: dce@smsc.Sony.COM (David Elliott)
Organization: Sony Microsystems Corp.
Lines: 48

Our site gets a pretty full newsfeed and has 250MB of space set aside
for news articles.

I've been trying to get the C news explist stuff set up for maximum
use of the space, but I am a little paranoid when it comes to raising
the expiration times because I've seen too many instances of news
logjams getting unstuck and filesystems overflowing.

I've seen programs that expire based on disk space, allowing you
to prioritize newsgroups and expire until there is a minimum amount
of free space, but this may come to late.

It seems to me that a better idea would be to have a program that
generates a list of files to remove in order of "removability".
When the unbatcher starts working, it would look at the space
available, and while there wasn't enough, it would remove files
from the top of said list.

This could even be used in concert with the current expiration
mechanism to allow for a general smooth removal of articles that
really are out of date.

The "removability" of a file would be a function of newsgroup
name, newsgroup size, and file age.  One might use a formula
like:

	removability = (X*size^2 + Y*age^2) - usefulness(newsgroup)

The "usefulness" function would be a table of constants supplied
by the administrator.  The values of X and Y are supplied for
each newsgroup to give weight to these items.

For example, the table entries

	# Group         Usefulness   Size   Age
	rec.music	        15      3     5
	rec		        10      1     1

This says that rec.music.* rates better than the other rec groups,
but that the newsgroup should be smaller, and the articles
become useless pretty fast.

Comments?
-- 
David Elliott
dce@smsc.sony.com | ...!{uunet,mips}!sonyusa!dce
(408)944-4073
"But Pee Wee... I don't wanna be the baby!"