Path: utzoo!utgpu!water!watmath!clyde!att-cb!osu-cis!tut.cis.ohio-state.edu!mailrus!ames!pasteur!ucbvax!hoptoad!gnu
From: gnu@hoptoad.uucp (John Gilmore)
Newsgroups: news.software.b
Subject: Re: news software speedup
Message-ID: <4246@hoptoad.uucp>
Date: 27 Mar 88 00:38:08 GMT
References: <649@bms-at.UUCP> <10150@ncc.UUCP>
Organization: Grasshopper Group in San Francisco
Lines: 34

lyndon@ncc.UUCP (Lyndon Nerenberg) wrote:
> There is some potential for trouble on System V implementations using
> this method. Most USG systems are configured with a ulimit of 1 meg.

I think this is considered a bug by one and all.  (At least by me!)
If we distribute a netnews release that requires that the
bug be fixed, the *&^$%# vendors who ship systems with this bug will
scramble to fix it.

Saying "we shouldn't speed up netnews with a new architecture because of a
System V bug" is like saying "we shouldn't run netnews in the current
architecture because a System V bug loses inodes when you do".  The
solution is to fix the bug, not to stop working on making netnews faster
and easier to manage.

I am pretty happy with the speed of "rnews" under C news.  Where I want
more performance is on "expire"; it takes hoptoad more than an hour to
run a single expire each night, and disk response on that drive goes to
hell.  I *think* the problem is in the dbm code.  Note that expire
spends most of its time handling entries for already-expired messages,
since typically the message-ID-retention is set to e.g. 28 days while
the expire threshold is at 14 or 10 or 8.  Nevertheless, it seems to grind
up the disk even while processing these, and all it's doing is reading
a line, doing an ftell(), writing the line out, and adding a dbm entry.

If somebody gets fired up to rewrite dbm in the public domain, please
make an "in core" option where it won't write to the disk at all, just
malloc the blocks it wants, and keep track of where it wants to put
them.  My dbm files are under 3MB total, and if I can keep most of that
in memory while building them, we should be able to zip right through
the expired stuff in no time, without moving the disk heads at all.
-- 
{pyramid,ptsfa,amdahl,sun,ihnp4}!hoptoad!gnu			  gnu@toad.com
		"Watch me change my world..." -- Liquid Theatre