Path: utzoo!mnetor!uunet!munnari!vuwcomp!duncan
From: duncan@comp.vuw.ac.nz (Duncan McEwan)
Newsgroups: news.admin
Subject: Re: Map expiration dates
Message-ID: <13226@comp.vuw.ac.nz>
Date: 26 Jan 88 23:18:56 GMT
References: <2323@cxsea.UUCP> <7815@rutgers.rutgers.edu>
Reply-To: duncan@comp.vuw.ac.nz (Duncan McEwan)
Organization: Comp Sci, Victoria Univ, Wellington, New Zealand
Lines: 112
Summary: Why post the full map at all?

In article <7815@rutgers.rutgers.edu> pleasant@rutgers.rutgers.edu 
(Mel Pleasant) writes:

>For those of you interested in lowering
>the volume, here is a challenge for you.  How can it be done such that we
>never have to make a full one-shot posting of all of the files ever again?

This definitely includes us, given the relatively high communications
charges of receiving usenet in this part of the world.

I originally mailed a response to this directly to Mel.  I never got
any feedback from my suggestions/queries, so I don't know if he
received it (though I guess more likely, he was just too busy to
respond).  I have waited a week or so to see if anyone else has said
anything similar but I haven't seen anything, so here goes...

>The problem with "diff" postings is [possibility of losing parts or
>parts arriving out of sequence].

I have supported the idea of posting diffs in the past.  I don't think
Mel's reasons against them are insurmountable.

If the diff listings contained sequence numbers you could tell if you
missed something.  If various sites around the net offered an
automated "send me map file n" service in much the same way that some
sites operate source archive servers, those that missed part of the
map could get the bits they missed.

[As an aside - even with the current map posting scheme it is not possible
to tell if you have missed something.  I think it would be a good idea if
whatever scheme is used, the map postings contained a sequence number in
the subject].

Anyway, a much more convincing argument against diff postings was in a
followup to Mel's original article by Brian Matthews in which he said
he had exerimented with diff, and on average found diff listings to be
larger than the new files!  A larger sample might be required to see
if this is generally the case.

A final (overwelming?) argument against diff postings is that they
would make much more work for all us overworked sys admins :-).  If a
posting is missed, we would have to manually intervene before
subsequent updates can be used.  For many, maintaining an up to date
map is not a top priority, and with the current scheme, we can choose
to wait until the next update, and put up with a slightly inaccurate
map for a while.

So assuming diff postings will not work, we (finally) get to the point
of this article.

>We are now posting on the 1st of each month those map files that
>have not been updated in 30 days.

This is the part that doesn't make sense to me (if people really want the
map posting volume reduced).  While the new posting scheme is great for
ensuring quick update propagation, it will obviously result in a volume
increase as long as this 30 day reposting is carried out.

So why is it required?

It might reduce the problem of people missing map postings.  But
they may still miss parts of the periodic repost so it doesn't
eliminate it altogether.

It also allows sites new to usenet to get a full copy of the map.  But
it is quite likely that their usenet neighbour has a copy of the map
that they can pass on.  And for those sites for which this is not the
case, a monthly posting listing sites that have the map available for
anonymous uucp should suffice.

>As you point out, the maps are read by a program and after that they
>aren't needed, right?  Well, wrongo!!  The number of people [...]
>complaining of loss of access to map files was astounding.  There are
>many [...] that actually use the map files to generate paths by hand.
>When the map files expire they're at a loss.

To satisfy these (backward :-) people without a periodic repost, why
can't the map's be posted with a long expiry date (sometime in the
21st century should do).

I have experimented a little with the effects of the Supersedes header
on articles that have such long expiry dates - checking in particular
that the history record for the original article isn't maintained for
ever (which would result in history files steadily getting larger).  As
far as I could tell, this was not the case.  Perhaps someone who knows
for sure could confirm/deny this.

Two other problems with long expiry dates spring to mind.

1) I believe in order to cut down on abuse of the expiry header some
   sites run expire with the options that cause it to ignore expiry dates.
   These sites would lose the map after their normal expiry period.

2) For sites that have not installed a news system that handles the
   "Supersedes: " header, the map directory will keep getting bigger.
   (with the current scheme they will temporarily have multiple copies
   of a particular map file, but eventually the older copies will
   disappear).

The solutions to both these problems are fairly obvious, and not that
unpalatable.  Others can decide whether they are worth a decrease in
map volume posting.  I think they are.

I am sure that without the periodic reposting of the entire map, the
current scheme would result in a reasonable map update volume.  If
this is not the case, perhaps some of the larger files could be
further split up to reduce the impact should they be modified too
frequently (though I doubt this will be necessary).

I would like to hear from anyone who can think of problems not mentioned
here, of either long expiry dates, or not having the periodic repost
of the unmodified map files.

---
Duncan

Internet: duncan@comp.vuw.ac.nz		Path: ...!uunet!vuwcomp!duncan