Path: utzoo!utstat!news-server.csri.toronto.edu!math.lsa.umich.edu!zaphod.mps.ohio-state.edu!wuarchive!emory!hubcap!ncrcae!ncr-sd!sceard!mrm
From: mrm@sceard.Sceard.COM (M.R.Murphy)
Newsgroups: news.software.b
Subject: Re: Cnews expire problem... need help
Message-ID: <1990Dec8.190114.15171@sceard.Sceard.COM>
Date: 8 Dec 90 19:01:14 GMT
References: <1990Dec7.130639.15803@bnr.ca> <660596702.10086@mindcraft.com>
Reply-To: mrm@Sceard.COM (M.R.Murphy)
Organization: Sceard Systems, Inc.  San Marcos, CA  92069
Lines: 100

In article <660596702.10086@mindcraft.com> karish@mindcraft.com (Chuck Karish) writes:
>In article <1990Dec7.130639.15803@bnr.ca> janick@bnr.ca
>(Janick Bergeron) writes:
>>For the last few days, I keep receiving the following message from
>>Cnews/expire:
>>
>>expire problems:
>>expire: wrong number of fields in `	660190899~-...'
>>
[...]
>
>New question: What causes the history lines to be mangled in
>the first place?  I'm getting about eight or ten of them a
>month.  Is it related to having my spool disk fill up?
>-- 
>
>	Chuck Karish		karish@mindcraft.com
>	Mindcraft, Inc.		(415) 323-9000		

In our case, it was a bad sector on the disk :-) :-(

The following code cleans up a history file so that mkdbm is happy with it,
and also replaces the single awk line that sifts a history file and prints
only lines that are after a given time that I used in a modified expire scheme.
The checking for goodness in a history line could be made fancier, but this is
enough to make mkdbm happy. Makes for a pretty fast expire, too. Every so often,
writing a short specialized tool in C is appropriate, though I'd rather use
awk :-) In the case of a bad disk block, awk groused about a record too long and
bailed. If 8192 is bad for a buffer here, somebody could get fancy with malloc,
or maybe just write the whole thing in one line of perl.

---- cut here ----
/*
 * exphist - scan history and write only good lines for mkdbm
 * usage is exphist time_from_getdate
 *
 * cursory examination of this code will show that it is snagged from mkdbm.
 *
 */
#include <stdio.h>
#include <string.h>

main(argc, argv)
int argc;
char *argv[];
{
	long		exptime;
	long		atol();
	static char	buff[8192];
	register char	*scan;
	register char	*line;

	if (argc < 2 || (exptime = atol(argv[1])) == 0) {
		fprintf(stderr, "Usage: exphist time\n");
		exit(2);
	}

	for (;;) {
		line = fgets(buff, sizeof(buff), stdin);
		if (line == NULL)
			break;
		scan = strchr(line, '\t');
		if (scan == NULL || line[strlen(line)-1] != '\n') {
			fprintf(stderr, "bad format: `%.60s'\n", line);
			continue;
		}
		if (atol(++scan) > exptime)
			fputs(line, stdout);
	}
	exit(0);
}
---- cut here ----

This is the fragment from expire:

...
now=`getdate now`
ago=`awk "/^\/expired\// {print ($now-(86400*\$(3)))} {next}" explist`
# replace the single-line awk with exphist
#awk "{split(\$2,dates,\"~\");if(dates[1]>$ago)print \$0}" history >history.n
exphist $ago < history >history.n
mkdbm history.n &&
[ -s history.n ] &&
mv history history.o &&		# install new ASCII history file
mv history.n history &&
rm -f history.pag &&		# and related dbm files
rm -f history.dir &&
mv history.n.pag history.pag &&
mv history.n.dir history.dir
...

Since expire will be executed from cron, the error messages from exphist
and mkdbm will show up in mail to somebody important should they occur.
Note the "&&" after the mkdbm step. This is added to keep history relatively
intact should there be a major failure.

None of this stuff takes too kindly to bad blocks, totally running out of
file system space, or inodes, or such, but then, what of UNIX(tm) does?
-- 
Mike Murphy  mrm@Sceard.COM  ucsd!sceard!mrm  +1 619 598 5874