Path: utzoo!utgpu!water!watmath!clyde!att-cb!att-ih!alberta!ubc-cs!van-bc!sl
From: sl@van-bc.UUCP (pri=-10 Stuart Lynne)
Newsgroups: news.software.b
Subject: Re: news software speedup
Message-ID: <1696@van-bc.UUCP>
Date: 26 Mar 88 22:37:53 GMT
References: <649@bms-at.UUCP> <10150@ncc.UUCP>
Reply-To: sl@van-bc.UUCP (pri=-10 Stuart Lynne)
Organization: Public Access Network, Vancouver, BC.
Lines: 48

In article <10150@ncc.UUCP> lyndon@ncc.UUCP (Lyndon Nerenberg) writes:
>In article <649@bms-at.UUCP> stuart@bms-at.UUCP (Stuart D. Gathman) writes:
>
>   A major problem with the current system is scanning article headers
>   in many seperate files.  (And unix doesn't like big directories to boot.)
>   My idea is to have 2 files per newsgroups directory (other than sub-
>   directories).  All headers for a news group would be in one file
>   with offsets into another file containing all articles.


>There is some potential for trouble on System V implementations using
>this method. Most USG systems are configured with a ulimit of 1 meg.
>If you exceed this limit within a particular newsgroup, articles will
>be lost when the append to the file fails.
>
>There are two solutions: 
>
>	1) Increase the ulimit by reconfiguring the kernel, or
>	2) Make rnews setuid to root, allowing it to increase
>	   its ulimit value.
>

Actually there is a third solution. The original suggestion was to store
headers in one file and all articles in a second file. The rationale being
to reduce significantly the use of Inodes, processing time for news readers
etc.

This could be modified to be n+1 files, the first file stores the headers,
plus offset *and* file name for each article. Instead of storing all
articles simply spread them among several (n) files. Several different types
of criteria could be applied to place the articles among the files:

	1. a new file everytime the 1MB ulimit was in danger of being broached
	2. a new file every day/week/arbitrary period
	3. multiple files based on keyword/subject/other sorting criteria
	4. a new file for each article

The fourth option is essentially status quo except for moving the history
file for /usr/lib/news/history to be distributed into the news spool
directories.

I do see some problems for cross posting. But certainly they wouldn't be
insurmountable.



-- 
{ihnp4!alberta!ubc-vision,uunet}!van-bc!Stuart.Lynne Vancouver,BC,604-937-7532