Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!uunet!husc6!cmcl2!nrl-cmf!ames!aurora!jaw
From: jaw@aurora.UUCP (James A. Woods)
Newsgroups: news.misc
Subject: Re: Compress binaries? Some real numbers.
Message-ID: <1306@aurora.UUCP>
Date: Tue, 17-Nov-87 21:23:28 EST
Article-I.D.: aurora.1306
Posted: Tue Nov 17 21:23:28 1987
Date-Received: Fri, 20-Nov-87 04:24:58 EST
References: <349@gethen.UUCP>
Distribution: all
Organization: NASA Ames Research Center, Mt. View, Ca.
Lines: 101

# Oh, the shark has pretty teeth, dear--
  And he shows them pearly white
  Just a jackknife has Macheath, dear--
  And he keeps it out of sight.	    [Bertolt Brecht, Threepenny Opera] 

Now that Michael Farren has shown the diminished role for ARC within
USENET, let's once again return to standard Unix tools.

[Aside:  However, the analysis is further complicated by realizing 
that Telebit modems now do "compress -b11" in firmware.
I presume that big-CPU sites feeding similarly capable machines
would favor the more efficient software compress code, and turn
the modem feature off.  But smaller nodes could use the setting,
and not worry too much when the modem is fed random bits, since
expansion would never happen if the modem double buffers.]

Though I'm not a binary-posting freak (Unix is supposed to free us from
these shackles, right?), maybe it's time to revive the idea of 'shark',
now circulating on Compuserve (and discussed in comp.sources long ago).

It does decompression, de-archiving, and binary restoration without
unpleasant argument over the format, because it's self-decoding.
The receiver only needs a shell, a C compiler, and 'tar' (or Gilmore's
PD 'tar').  Last I heard, Macs, Amigas, Ataris and MS-DOS boxes all
qualified.  Warning: it does the compression *after* archiving (much
easier), so Farren's statistics might not be the same.  The decoder
length is short enough to amortize the xmission benefit at 10KB or so.

A simple Unix-only application of 'shark' is a mailer which splits up
large compressed packages for sending, where the recipient just
accumulates pieces with the mail 'w' command, then runs 'sh': 

	#!/bin/sh
	m=$1; shift
	shark $* | split -800 - /tmp/shark$$
	n=`ls /tmp/shark$$* | wc -l | sed 's/  *//'`
	p=0
	for f in `ls /tmp/shark$$*`
	do
		p=`expr $p + 1`
		mail -s "bundle from '`whoami`' ($p of $n)" $m < $f
	done
	rm /tmp/shark$$*

(The 'split' guarantees pieces < 64K.)   Usage: "sharkmail address files..."
Again, because of the statistics controversy, don't use it for sending
files to USENET, unless you're sure of the total system effect.  In case
you've forgotten, here's the rest:

------------
echo '#!/bin/sh
#"Copyright (C) 1988 by James A. Woods.  All rights reserved."
#"Cleverly he dialed from within." -- D. Van Vliet, "Trout Mask Replica"
PATH=$PATH:.
trap "rm -f zcat atob az$$*;exit" 0 1 2 3 15
echo decoding...
cat>az$$.c<<\_'
cat<<'TROUSER PRESS'
#include<stdio.h>
#define d define
#d I if
#d E else
#d W while
#d C char
#d L long
#d U unsigned
#d B(n) (1<<(n))
#d K bcount
#d G getchar()
#d P putchar
#d X return
#d H(n) ((C*)h)[n]
#d V(a,b) (B(b)-1&a)
#d Q 256
#d Y w=w*85+c-33,
#d Z w=Q;W(w--)t[w]=0
#d x break;
#d F {X(write(2,"shar botch: resend\n",19));}
L K,o,v,w,c,m,M=B(16),f;int i,q,b,n,k=16,e=128,j,O,S;U short t[69001];FILE*T;U
L h[69001];C D[255];d(c)int c;{I(c=='z'){I(K)F E{c=4;W(c--)y(0);}}E I(c>32&&c<
118){I(!K)w=c-33,++K;E I(K<4)Y++K;E Y y(w>>24),y(w>>16),y(w>>8),y(w),w=K=0;}E
F;}y(l)L l;{c=(int)(l&255);o^=c;m+=c;m++;I(1<<31&f)f*=2,f++;E f*=2;f+=c;putc(c
,T);}main(g,v)C**v;{int c;L i,A,h,y,s,r;I(**v!=97)X(a());sprintf(D,mktemp(
"azXXXXXX"));I(T=fopen(D,"w+"))unlink(D);E F;W(1){I(!fgets(D,255,stdin))F;I(!
strcmp(D,"xbtoa Begin\n"))x}W((c=G)>=0){I(c==10)continue;E I(c=='x')x E d(c);}
I(scanf("btoa End N %ld %lx E %lx S %lx R %lx\n",&A,&h,&y,&s,&r)!=5)F;I(A!=h||
y!=o||s!=m||r!=f)F E{fseek(T,0L,0);i=A;W(i--)P(getc(T));}}L g(){C*p=D;I(j>0||O
>=S||f>m){I(f>m)m= ++n==k?M:B(n)-1;I(j>0)m=B(n=9)-1,j=0;S=fread(D,1,n,stdin);I
(S<1)X-1;O=0;S=S*8-n+1;}q=O;b=n;p+=q>>3;q&=7;c=V(*p++>>q,8-q);q=8-q;b-=q;I(b>7
)c|=(*p++&255)<<q,q+=8,b-=8;c|=V(*p,b)<<q;O+=n;X c;}a(){I(G!=31||G!=157)F;k=G;
e=k&128;k&=31;M=B(k);I(k>16)F;z();}z(){L w;C*s;m=B(n=9)-1;Z,H(w)=(C)w;f=e?257:
Q;i=o=g();I(o==-1)X;P((C)i);s=(C*)&H(B(16));W((w=g())>=0){I(w==Q&&e){Z;j=1;f=Q
;I((w=g())==-1)x}v=w;I(w>=f)*s++=i,w=o;W((U L)w>=(U L)Q)*s++=H(w),w=t[w];*s++=
i=H(w);do P(*--s);W(s>&H(B(16)));I((w=f)<M)t[w]=(U short)o,H(w)=i,f=w+1;o=v;}}
TROUSER PRESS
echo '_
(set|compress|btoa|atob|zcat)2>/dev/null 1>&2||
(cc -o zcat az$$.c;ln zcat atob)
(atob|zcat|tar xBvf -)<<\TROUSERS'
tar cBf - $*|compress|btoa
echo TROUSERS