Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!mnetor!uunet!seismo!sundc!pitstop!sun!decwrl!ucbvax!ucsfcgl!cca.ucsf.edu!ccb.ucsf.edu!dick
From: dick@ccb.ucsf.edu (Dick Karpinski)
Newsgroups: comp.misc
Subject: Re: Faster/cheaper execution of Unix pipelines?  A proposal.
Message-ID: <1062@ucsfcca.ucsf.edu>
Date: Mon, 2-Nov-87 20:24:30 EST
Article-I.D.: ucsfcca.1062
Posted: Mon Nov  2 20:24:30 1987
Date-Received: Fri, 6-Nov-87 21:20:18 EST
References: <385@ohlone.UUCP>
Sender: root@cca.ucsf.edu
Reply-To: dick@ucsfccb.UUCP (Dick Karpinski)
Organization: UCSF Computer Center
Lines: 43
Keywords: Unix pipes, program composition, small is beautiful
Summary: Jackson gives one way, Modula-2 another, or roll your own.

In article <385@ohlone.UUCP> nelson@ohlone.UUCP (Bron Nelson) writes:
>... discussion ...
>"Big Programs Hurt Performance" and the relative merits of a single
>program with lots of options, vs. communicating small programs,
>in particular the use of pipes in the Unix world.  
>...  The software engineer in me insists that "many small
>routines, each doing one job well" is the correct thing to do, but
>my practical side recognizes the high cost of Unix pipes.  
>... the answer is to invent a cheaper method of combining ....

In Principles of Program Design, Michael Jackson shows exactly how
to do that by flattening the control structure of each piece.  In
Modula-2, Niklaus Wirth (with "light weight" processes) shows how
to do it in general (except for passing the arguments).  James O.
Baker, in a private communication to me, explains how to do the
argument passing.  This reduces the cost of pipes to (roughly) the
cost of calling a subroutine.  The general notion is called
"co-routines" in the computer science literature.  My cut at it
is to enhance the linker/loader to be able to compose alternative
implementations of modules in a "streams"-like fashion.

In my version, the interface into any module (such as the collection
which supports standard output) is an appropriate place to install a
programatic filter.  I call it the extender-board-module notion.  It
inserts a (conceptual) extender board with a prototyping area between
an existing client and its support module.  A pipe then is simply an
output acceptor for the left routine and an input provider to the
right routine.  There are other places where a well modularized
program can profitably be split apart for one purpose or another.
The term programatic filter is probably ill chosen since all Unix
filters are programs, but is intended to distinguish it from the
simple byte stream filter, since most modules are more complex.

I, too, welcome comments about where these ideas have already been
implemented or demonstrated.

Dick

Dick Karpinski  Manager of Minicomputer Services, UCSF Computer Center
UUCP:  ...!ucbvax!ucsfcgl!cca.ucsf!dick        (415) 476-4529 (11-7)
BITNET:  dick@ucsfcca or dick@ucsfvm           Compuserve: 70215,1277  
USPS:  U-76 UCSF, San Francisco, CA 94143-0704   Telemail: RKarpinski   
Domain: dick@cca.ucsf.edu  Home (415) 658-6803  Ans 658-3797