Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Posting-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!henry
From: henry@utzoo.UUCP (Henry Spencer)
Newsgroups: news.software.b
Subject: Re: Proposal: compressing news in the spool directories
Message-ID: <7923@utzoo.UUCP>
Date: Fri, 17-Apr-87 18:03:24 EST
Article-I.D.: utzoo.7923
Posted: Fri Apr 17 18:03:24 1987
Date-Received: Fri, 17-Apr-87 18:03:24 EST
References: <536@vixie.UUCP>
Organization: U of Toronto Zoology
Lines: 33
Keywords: proposal, compressed news, storage

> What about compressing the news data in the spool directories? ...

Apart from the issues of cpu time consumed, the slowdown in reading news,
etc., remember that compress does not do nearly as well on short files as
it does on long ones.  Most news articles are quite short.

> ... If compress
> can save half the transmission time, it ought to be able to save almost that
> much in the storage costs as well...

A quick experiment suggests that you'll be lucky to get 25% consistently.
Perhaps a smarter data-compression algorithm could do better, but *I* don't
want the job of inventing one.

> This may all be moot, given the onset of C news.  I don't know if C news
> does this or not.

Nope.  Geoff and I talked about this and other issues for a while.  After
considerable discussion and thought, we concluded that we did not know any
way of storing news articles that would be *decisively* superior to the
current one.  Most of the ideas we kicked around had disadvantages as well
as advantages, and we couldn't see anything that was enough of a net win to
be worth the trouble of changing over.

The specific problems of storing compressed articles (apart from the issue
of header searching, which isn't needed by C expire but would be significant
for the news readers) are the very modest saving in space for small files
when using current compression algorithms, the awkwardness of applying
general Unix tools to non-ASCII files, and above all the large overheads
it would add to news reception and reading.
-- 
"We must choose: the stars or	Henry Spencer @ U of Toronto Zoology
the dust.  Which shall it be?"	{allegra,ihnp4,decvax,pyramid}!utzoo!henry