Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!uwm.edu!uakari.primate.wisc.edu!gem.mps.ohio-state.edu!tut.cis.ohio-state.edu!purdue!bu-cs!mirror!ima!cfisun!lakart!dg
From: dg@lakart.UUCP (David Goodenough)
Newsgroups: news.admin
Subject: Dynamic feeds (was Re: Why not just eliminate  ..... )
Message-ID: <725@lakart.UUCP>
Date: 18 Oct 89 16:15:09 GMT
References: <817@pmafire.UUCP>
Organization: Lakart Corporation, Newton, MA
Lines: 83

I've posted this only to news.admin, because the direction it's
taking it has left the newsgroup name thread, and become an admin
subject. I'm also putting on my +6 ring of fire resistance, because
I'm going to speak my mind, and probably offend a lot of people.

geoff@pmafire.UUCP (Geoff Allen) sez:
> In article <34075@looking.on.ca> brad@looking.on.ca (Brad Templeton)
> proposes a scheme for setting up dynamic feeds
> 
> I think this idea's worthy of consideration.  Just how tough would
> something like this be to do?

I'd make one suggestion, partly because I believe this is the way to
write software, and partly because it makes this job easier.

Break up inews.

I sometimes get the feeling inews is too big for it's own good. Inews
handles many functions:

posting user articles;
unbatching incoming news;
manipulating control messages; (several functions in this alone)
preparing data for "ihave - sendme" runs.

What is wrong with making these separate programs. I can't believe that
it's disk space: These separate programs might in total be bigger than
inews, but they'd be easier to maintain, install and work with. I started
out on a PDP11 34A (you know, the ones where each process gets a gallon
of ram tops) That approach has stayed with me, as a result I keep 'em
small.

Back to the separate program issue. If this were done, then addition
of Brad's idea would be trivial, and could be added incrementally to an
already installed setup. The one question I have is topology - it sounds
to me like this relies a bit on an acyclic tree. Well, we don't got that.

I'd suggest more a dynamic "ihave - ineed" type setup. I.e.

1. generate the list of newsgroups that you want (the problem here is
transmission time - see the note below)

2. generate an "Ihave" list from those newsgroups, this could be done with
grep and tr and a shell script

3. send the Ihave / newsgroup list to your first feed (one at a time)

4. your feed builds HIS Ihave list for you, sends it back to you, along
	with any articles you need (from his point of view)

5. you figure which articles your feed needs from you, and send them off.

6. At this point, if you have multiple feeds move to the next feed, and loop
back to step 2.

A few issues need to be considered.

1. What happens if another feed calls _YOU_ while this is running - the system
_MUST_ be able to run as slave for multiple systems concurrently. I think this
may work anyway, since in slave mode you are just processing a request list.
Think about UUNET :-/

2. How do we solve the cyclic tree problem. I get news from both cfisun and
mirror, I would consider cfisun to be a feed, but what about mirror. Are they
a feed for us or are we a feed for them. Really the problem becomes how to
pass newsgroup information "up the tree" - if mirror decide that we feed them
and turn round and ask for every newsgroup on their system we're gonna flip
out: we only have a 2400 BPS modem for communications.

3. how about a system that is five hops down the tree from a backbone. It
would take ten days (assuming one run per day) for a newsgroup appearing
in my .newsrc to start getting news: five days for the information to
percolate to the top, and five more days for the data to get back down to
me. I grant this is not realistic: two hops == four days is more like it,
but it's still a while.

I agree with Brad - this has to be possible (watch out for D news :-) ), but
there are going to have to be some questions answered first.
-- 
	dg@lakart.UUCP - David Goodenough		+---+
						IHS	| +-+-+
	....... !harvard!xait!lakart!dg			+-+-+ |
AKA:	dg%lakart.uucp@xait.xerox.com			  +---+