Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP Path: utzoo!mnetor!seismo!lll-crg!lll-lcc!qantel!hplabs!ucbvax!ucbarpa.Berkeley.EDU!fair From: fair@ucbarpa.Berkeley.EDU (Erik E. Fair) Newsgroups: net.news,net.micro.mac Subject: Re: Backbone automatic news-compression question ... Message-ID: <15746@ucbvax.BERKELEY.EDU> Date: Sun, 21-Sep-86 04:55:58 EDT Article-I.D.: ucbvax.15746 Posted: Sun Sep 21 04:55:58 1986 Date-Received: Sun, 21-Sep-86 22:02:51 EDT References: <4012@ut-ngp.UUCP> Sender: usenet@ucbvax.BERKELEY.EDU Organization: University of California at Berkeley Lines: 64 Keywords: question [regarding] compression [of] news [for] transmission Summary: Yes, the backbone uses compression; so does every other sensible admin Xref: mnetor net.news:2039 net.micro.mac:7108 Included in the netnews distribution since last year has been a public domain implementation of the Lempel-Ziv compression algorithmn, described in the June 1984 issue of IEEE Computer magazine, called "compress.c". Its maintenance has been taken over by a group of data compression hackers who use the "mail.compress" mailing list to facilitate the exchange of ideas on data compression. The compress program is currently in release 4.0, and to the best of my knowledge, 4.0 compress knows how to uncompress files compressed by all its previous versions... The basic method is that netnews will, for the purposes of transmitting an article to a site, append the filename of the article (e.g. /usr/spool/news/net/general/200) to a file that represents what site you want to send the article to (e.g. /usr/spool/batch/decvax). Periodically, a shell script is run that collects all the files (i.e. articles) listed in the file-of-files into a "batch" that is, a single file containing articles separated by standard markers. This batch is then compressed using the "compress" program supplied in the distribution, with whatever parameters that the two sites invovled have agreed upon (certain sites, like PDP-11's and 80x86 systems can't use full blast compress because of address space limitations, and for those sites, the sending site has to give compress an argument indicating that it should not go hog wild on memory use). Finally, the script will hand off the batch to the transport mechanism (usually UUCP) for queueing and subsequent transmission to the neighboring site. The neighbor site then reverses the procedure (uncompress, unpack the batch, process the articles) when the news gets there. On the sites that I run or have set up, not all links are compressed, but all links are batched. When I compress, I try to build batches of 120K bytes (on the assumption that I will get 50% compression), yielding a 60K file to send to my neighbor site; this is a magic number in that it takes approximately 10 minutes to transmit a 60K file at 1200 baud using UUCP, and this is a good granularity if there is any trouble with the link because UUCP overhead will only beat hard on me every 10 minutes, but if I lose the link (Telco gets noisy, or either computer crashes), I have at most 10 minutes of re-transmission to do. For those who are skeptical of the savings that compression gives, it was proven by (I think) Jerry Aguirre of Olivetti that what compress takes in CPU cycles is more than made up for in the savings of UUCP transmission CPU cycles (to say nothing of what it will do for your phone bill). Drop a note to him for the exact figures. As for the answer to Werner Uhrig's question specific question of whether compressing and BinHex'ing a mac thing would defeat the data compression that we use at transport; I doubt it, because as I understand it the process of BinHex'ing expands the object into printable ASCII again, much like UUUENCODE, which is eminently compressable (UUCP transport works with 8 bit channels, and handles binary data fine, it's just news that wants to play with 7 bit text). However, 1. I'm not one of the data compression hackers who really would know for certain. 2. When the difference is only 5K, my opinion is that it is not worth the trouble to compress at user-level. "The future's so bright, I gotta wear shades..." - Tim Buk III Erik E. Fair ucbvax!fair fair@ucbarpa.berkeley.edu