Xref: utzoo ont.uucp:341 tor.news:105 Newsgroups: ont.uucp,tor.news Path: utzoo!utstat!geoff From: geoff@utstat.uucp (Geoff Collyer) Subject: Re: genat down with disk troubles (Lsuc problems too) Message-ID: <1988Jan5.011219.2676@utstat.uucp> Summary: the storm was real; caused by B news on utzoo & lots of traffic Organization: Statistics, U. of Toronto References: <1987Dec23.161042.2776@utstat.uucp> <360@spectrix.UUCP> <1988Jan3.201804.10833@gpu.utcs.toronto.edu> <236@syntron.UUCP> Distribution: ont Date: Tue, 5 Jan 88 01:12:19 EST I am the news administrator for utzoo during Henry's absence. Just to set the record straight, there was a news storm or flood originating at utzoo at Christmas time. It has passed except that dciem hasn't picked it all up yet and genat has been down, or at least not ready to receive news, and so hasn't seen it yet. (So all you sites downstream of dciem and genat can gird your loins now: dciem is now receiving some news and genat appears to be healing. :-) I think there is a lesson (or two) in the story of the flood... Until mid-December, utzoo ran B 2.10 rnews [gasp! well it is public information; you could have discovered this by sending a version control message], but only from 6:30 PM each week night until about 8 AM the next morning, and around the clock on weekends, to prevent interference with real work during the day. Around the end of November, Henry noticed that not only was B rnews not processing all of the nightly news flow by 8 AM, but the 24-hour processing on the weekends left some unprocessed news on Monday morning. I did some gross measurements in early December which suggested that the rate of accumulation of unprocessed news was slightly over 1Mb/day. As you might expect from that rate, within three weeks of Henry first noticing a backlog on Monday morning, there was a backlog of about 25Mb of incoming news on utzoo by mid-December, growing rapidly. On December 15th, Henry installed C rnews and let it loose (subject to the same time-of-day restrictions as above) on the backlog at 6 PM. During the night of December 15th-16th, C rnews processed all but a few megabytes of the incoming queue and at 10:03 PM on December 16th, C rnews polished off the last article of the backlog. There's a lesson here for any other VAX-750-class machines (utzoo is a PDP-11/44 with 2 Eagles and 3Mb or 4Mb main memory) which don't process news during the day: Switch to C news. Soon. It's no fun putting up news software when there's a gun (consisting of a rapidly-growing backlog) pointed at your disks (but it certainly does concentrate the mind! :-). Thus endeth the advertisement. Then there was the small problem of distributing all that news. Just filing it all reduced utzoo's free space on /usr to 6Mb, at which threshold utzoo's news batcher refused to generate batches due to the lack of free space, and Henry left for vacation (December 21st). I spent a couple of days removing files, forcing calls to sites with dead autodialers, and waiting for the storm to expire. Around Christmas, expire started producing a lot of free space, so that batching could proceed even faster (now that the articles being batched had mostly expired :-). utstat gets a fairly small (and shrinking) subset of the available news, so I don't have a good feeling for how many megabytes are in a day's news flow, but I gather it is now about 1.5Mb-2Mb. A permanent, standing 1200 baud UUCP baud connection can pump no more than 9.5Mb/day (assuming 110 bytes/second, which is the empirical upper bound at 1200 baud). A surprising number of utzoo's news neighbours in Toronto have only 1200 baud modems, so this is not entirely academic. Looking a little into the future, keeping 9.5Mb/day for 10 days (as utzoo currently does) will consume 95Mb under /usr/spool/news and a site-specific volume in the outgoing UUCP queues. You will also need about 5Mb free for the incoming, unprocessed news for a single day (compressed). Let's look a little further into the future. B rnews on a VAX 750 under 4.2BSD processed about 67 bytes/second. C rnews, when I last measured it (over a year ago), was processing over 1,000 bytes/second on the same machine, so C rnews should not present a limit to volume of news until major news links use Telebit Trailblazers. Assuming 1,000 bytes/second through the Trailblazers and standing connections, one can only pump 86.4Mb/day. Retaining news for 10 days will consume 908Mb (864Mb in /usr/spool/news + 44Mb incoming), or 2.4 Sun Eagles, or about 2 Swallows. Somewhat later, CSRI should have a good FDDI Internet connection, so we should be able to transfer 10Mb of news per second, but C rnews will likely be running at only several kilobytes/second, unless we use a Cray as the main U of T news server. Unfortunately current disks typically transfer data no faster than about 2.5Mb/second, but we shall assume that disks will get magically faster. Assuming that the C rnews on the Cray can keep up with FDDI transfer rates, we can transfer only 864Gb of news per day. Keeping it for 10 days will consume 9Tb in /usr/spool/news, which will have to be on fast optical disks or in Cray 4 main memory. :-) :-) More seriously, I am interested in the growth of Usenet traffic vs. the increases in speed of news hardware and software, and may try to plot the curves. I would guess that in a few years, the growth of traffic will exceed the ability of machines smaller than Sun 4s running C news to keep up. I suspect that eventually only relatively large machines will be able to keep up with traffic volumes, especially if the owners of machines carrying news want people to get work (other than news maintenance) done on those machines. I see some final lessons: don't volunteer to be a backbone site :-); and only moderated groups will survive in the long-term. I wonder how long is "long-term": three years or five? I can remember news flows of 100kb/day; it all seemed so harmless then :-). -- Geoff Collyer utzoo!utstat!geoff, utstat.toronto.{edu,cdn}!geoff