Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!uunet!ukma!rayssd!galaxia!dave
From: dave@galaxia.zone1.com (David H. Brierley)
Newsgroups: comp.sources.wanted
Subject: Re: Looking for a netnews archiver program
Message-ID: <410@galaxia.zone1.com>
Date: Sat, 5-Dec-87 22:15:28 EST
Article-I.D.: galaxia.410
Posted: Sat Dec  5 22:15:28 1987
Date-Received: Fri, 11-Dec-87 04:40:24 EST
References: <303@caus-dp.UUCP> <61@icus.UUCP> <705@vaxine.UUCP>
Reply-To: dave@galaxia.zone1.com (David H. Brierley)
Organization: Dave's Very Own Personal System
Lines: 62

In article <705@vaxine.UUCP> wmp@vaxine.cs.ulowell.edu (Wayne Power) writes:
>In article <61@icus.UUCP> lenny@icus.UUCP (Lenny Tropiano) writes:
>>In article <303@caus-dp.UUCP> marcos@caus-dp.UUCP (Marcos R. Della) writes:
>>>I am looking for any program out there that will selectivly archive and
>>>compress news articles that come over the net, taking the sections to
>>
>
>I've considered writing a program that would eat news articles and put them
>into a news archive, leaving some header information in a data base so users
>could peruse the archives and select articles for extraction.  Given my work
>load, I wouldn't hold my breath.
>
>To echo Marcos' query, has anybody out there got one?

To answer the query, yes.  I have been working on exactly this problem on
and off for a while now.  My original intention was to create a mail based
archive server for use by the network community.  Unfortunately, my manager
decided our phone bills were too high and told me not to continue the project.
The project was recently brought back to life to allow archive access for
people within the company.  There is a possibility of opening up access to
the archives in the future but I wouldnt count on it.  What I can do is
release the software once it's complete.  A brief description of the software
follows:

- When articles are received for the specified groups (controlled by the
  sys file) they are placed into the archive.  The article itself is
  compressed and stored as YY.MM/NNN, where YY, MM, and NNN are the year,
  month, and unique id number.  The id number, article size, author, and
  subject are stored in a log file called GROUP/YY.MM where GROUP is the
  name of the newsgroup.  For cross-posted articles, a log file entry is
  made for each group.

- A mail-based archive server program is also provided which can return a
  canned help message, a list of the groups being archived, an index listing
  of a specified group, or a specific article.  The newsgroup index listing
  can be qualified by either author or subject (or both) using regular
  expressions.

The server program can be run in either an unrestricted mode or a restricted
mode.  In restricted mode, in order to retrieve an article the user must be
listed in a special validation file.  This was required for my environment
since I am forced to restrict access to people within the company but it's
very hard to prevent people from mailing to any address (or alias) that they
want to.

Anyway, the software is about 98% complete.  If anyone wants it let me know.
If there is enough demand I can post it.

Oh, one last thing.  In order to maintain the capacity of our disks, the
program allows previous months archives to be moved to tape and will
(semi-) automatically fetch an article from the tape if someone sends in
a reuquest for it.  The semi-automatic tape retrieval is the only part
that isn't completely working yet.

I know that other sites have various forms of archiving software but I am
partial to this one for several reasons.  First, the archive is maintained
compressed to conserve disk space.  Second, for the most part it is
completely automatic and requires very little attention.  Third, I wrote it.
-- 
David H. Brierley
Home: dave@galaxia.zone1.com	{cbosgd,gatech,necntc,ukma}!rayssd!galaxia!dave
Work: dhb@rayssd.ray.com	{cbosgd,gatech,necntc,ukma}!rayssd!dhb