Xref: utzoo comp.sources.bugs:2916 comp.compression:576
Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!swrinde!cs.utexas.edu!sun-barr!rutgers!cmcl2!kramden.acf.nyu.edu!brnstnd
From: brnstnd@kramden.acf.nyu.edu (Dan Bernstein)
Newsgroups: comp.sources.bugs,comp.compression
Subject: Re: Problem with compress
Message-ID: <12130:May1518:43:0491@kramden.acf.nyu.edu>
Date: 15 May 91 18:43:04 GMT
References: <26085:May1416:52:3491@kramden.acf.nyu.edu> <1991May14.185447.2498@nntp-server.caltech.edu> <1991May15.062504.28369@looking.on.ca>
Organization: IR
Lines: 28

In article <1991May15.062504.28369@looking.on.ca> brad@looking.on.ca (Brad Templeton) writes:
> It's a tough problem, actually.   It is nice to take the software tools
> approach of splitting archiver and compressor, but this makes it more
> difficult to do some of the cute things you can do when you combine them.
> The simplest of which is the fairly standard archive of compressed
> files, with each file compressed individually and extractable without
> decompressing the entire archive -- a serious problem if it is a
> multi-media archive or very large.

This is really getting away from comp.compression into
comp.unix.programmer, but anyway: All you have to do to make this work
is define a format for encoding multiple files into one stream, then
have your compressor (and, while you're at it, all your other tools)
work on such a stream. Let's say, for instance, that you use cpio
format, and have compress -f understand that format. Then you just pipe
cpio through compress -f and poof! all the files are individually
compressed.

I don't think cpio (or tar or extended cpio) is a particularly good
format for this: it's too restricted for some tasks (e.g., when you want
to multiplex two arbitrary-length streams) and too complex for others
(e.g., when you don't need or want filenames). So I've been working on
yet another format and a set of routines for applications to use that
format. Once it's done it'll solve a lot of old problems: grepping
through compressed files, for instance, or compressing each file in an
archive individually.

---Dan