Path: utzoo!attcan!uunet!mtndew!friedl
From: friedl@mtndew.UUCP (Steve Friedl)
Newsgroups: comp.unix.i386
Subject: Re: Compressing to a tape drive
Summary: Even quicker than dd...
Message-ID: <371@mtndew.UUCP>
Date: 14 Mar 90 05:27:14 GMT
References: <14496@s.ms.uky.edu> <1383@lzsc.ATT.COM>
Distribution: na
Organization: Steve's Barnburner 386
Lines: 57

It was suggested that this be used to send compressed data to a tape drive:

> find . -print | cpio -oc | compress | dd [blocking options] > /dev/rmt/c0s0

And HC Johnson suggests:
>
> If you want your tape to screem ( in bursts) try:
> dd of=/dev/rmt/c0s0 of=512b
> this will output 512 BLOCKS of 512 at a time.  The tape will run continuously
> for the big write, then rest while the buffer is refilled by the input to dd.

The adventurous might want to investigate another approach that
might help the tape stream even more.  The problem with the huge
blocksizes is that it stops while the buffer refills, so why not
use two processes sharing (at least) two buffers.

The parent process reads into some huge buffer, then forks a
child to write the buffer to the output place (the tape or
whatever).  While the child is writing the buffer, the parent is
busy filling a second buffer.  Once the second one is full, the
parent waits for the child to finish, then forks again to let it
write.

This does involve a process switch each block, but if the blocks
are large enough it might not be too bad.  Of course, you've got
to have copy-on-write VM or this become ridiculous really
quickly.

I have not actually done this, but I bet a donut that AT&T has.
Their "ctccpio" command feeding the stupid 23MB tape unit on the
3B2 has lots of process switches, and I think this is how they
manage to keep the tape streaming.  It trashes the machine for
the duration, but it is >much< faster than a dd with even a large
blocksize (it could also be because ctccpio turns on stream mode
with some ioctl).

You could probably do an even better job on this with a reader
and a writer with a common set of shared-memory buffers.  Then
you get the speed of double buffering without the process
switches.

While writing this article I started diddling with some actual
code to handle the process-switch part.  After ten minutes it was
just about finished, and I might post it in a few days if I ever
get it working.  I suppose one could really get a big win by
building the shared-memory version right into compress, so there
was no extra pipeio going on.

Thoughts, anybody?

     Steve

-- 
Stephen J. Friedl, KA8CMY / Software Consultant / Tustin, CA / 3B2-kind-of-guy
+1 714 544 6561 voice   /   friedl@vsi.com   /   {uunet,attmail}!mtndew!friedl

"How could anybody look at Miss April and *not* believe in a God?" - me