Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP Path: utzoo!mnetor!uunet!husc6!bbn!rochester!udel!burdvax!bpa!cbmvax!vu-vlsi!devon!ncoast!allbery From: tcjones@watdragon.waterloo.edu (Crocodile Dundee) Newsgroups: comp.sources.misc Subject: pipe - multi-input multi-output pipelining. Message-ID: <5322@ncoast.UUCP> Date: Sun, 8-Nov-87 16:40:16 EST Article-I.D.: ncoast.5322 Posted: Sun Nov 8 16:40:16 1987 Date-Received: Wed, 11-Nov-87 05:04:25 EST Sender: allbery@ncoast.UUCP Organization: U. of Waterloo, Ontario Lines: 294 Approved: allbery@ncoast.UUCP X-Archive: comp.sources.misc/8711/5 And now, a truly miscellaneous thing, a shell script to do multi-input multi-output pipelining. "An invaluable tool that you will probably never use." Terry. #!/bin/sh # This is a shell archive. Remove anything before this line, # then unpack it by saving it in a file and typing "sh file". # # Wrapped by watdragon!tcjones on Sat Nov 7 04:51:20 EST 1987 # Contents: pipe_doc pipe echo x - pipe_doc sed 's/^@//' > "pipe_doc" <<'@//E*O*F pipe_doc//' Here is a thing for those times when the shell input/output facilities just wont let you do what you want to in any simple manner. Basically it's a multi-input multi-output pipe command. Call it "pipe". The idea is that the input is sequentially assembled from possibly several sources and written (at possibly different times during the assembly) to possibly several destinations. The input sources can be the terminal or the output of any other command - even the output of another "pipe". The output can be to the terminal, or to any normal command - including a normal pipeline, or to the input of another "pipe". The syntax is easy. Command line arguments are one of three things, 1) the word "-in" 2) the word "-out" 3) a command -in switches you to "pipe input mode", -out switches you to "pipe output mode". A pipe always begins in input mode. An example might make this clear... % pipe -in "echo hello" -out "|wc" 1 1 6 Which is to say, "Go into input mode, take input from the command 'echo hello' (which of course produces the single word 'hello'), switch to output mode, and send the output so far into the command '|wc'" Notice that the commands above ("echo hello" and "|wc") must be passed to pipe as a single argument. Thus it is necessary to enclose multi-word commands in quotes. Also it is necessary to escape characters such as "|" and ">" from the shell. "" quotes do both of these jobs nicely, while still allowing for the expansion of environment variable and commands in backquotes. "-in" and "-out" can be abbreviated to "-i" and "-o" respectively and so this could have been written as pipe -i "echo hello" -o "|wc" or, (since we start off in input mode) as pipe "echo hello" -o "|wc" Now this could have been done easily with "echo hello | wc" using a normal one-in, one-out pipeline. More sophisticated examples will be coming up. To take input from standard input, or write to standard output, use a "-" So, % pipe - -o - hello there ^D # have to close standard input with EOF hello there % does the same thing as "cat" (alone on a line) would have done, reading from the terminal and writing back to the terminal. If you immediately go into output mode, pipe will assume you want to provide initial input from standard input, so the above could be written as "pipe -o -". If you don't give an output directive, pipe will assume you want standard output, so the last example could have been written simply as "pipe". That's about all there is to know, here are some examples % pipe - "grep tcjones /etc/passwd" - -o "|mail tcjones" - Here is my entry in /etc/passwd ^D Hope it's what you want. Terry. ^D % Here is my entry in /etc/passwd tcjones:Wwo5wq.gu/wd.:134:113:Crocodile Dundee,MC6129x6687,8843852,88463 38,103 minota hagey residence:/u2/tcjones:/usr/public/itcsh Hope it's what you want. Terry. % Which takes a line from my terminal, concatenates the output of grep, concatenates another line from my terminal, sends it all off in mail to me, and puts the output on the screen as well. Decidably more useful than the earlier examples. Here's another, % pipe - "cat last_lines" -o "|sort >sorted" ">unsorted" "|wc -l" here is a sample line here is another and a third ^D 32 % which takes lines from my terminal, concatenates the contents of the file last_lines, sends all that into sort (which sorts the lot and leaves the output in the file "sorted"), puts a copy in the file "unsorted" and also sends the lot to "wc -l" so I can see how many lines there were. (The last_lines file already contained 29 lines) It is always possible to omit the "-" command from a "-o" or "-i" section, provided that there is no other command in this section - which is to say that the section must be left empty. It is possible to switch back and forth between input and output modes, % pipe -o "|wc" -i -o "|wc" -i -o "|wc" this ^D 1 1 5 is ^D 2 2 8 the first ^D 3 4 18 % which takes lines from the terminal until an EOF, sends them to "wc", switches back to input mode (note there are no commands and so standard input will be used) to read more lines from the terminal, sends the current input to "wc" and so on and on... To have the whole thing sent to the screen at the end I could have appended a "-" OR used another "-o" with no command. Notice that the output of wc in each case reflects the cumulative way in which the input is being assembled. As a final example, you can intermix pipes with unix pipelines and i/o redirection to create complicated things like % pipe "echo this" | pipe - "pipe 'echo is'" | pipe - "echo dumb" this is dumb % Which could be used to do more useful things. That's probably enough... @//E*O*F pipe_doc// chmod u=rw,g=,o= pipe_doc echo x - pipe sed 's/^@//' > "pipe" <<'@//E*O*F pipe//' #!/bin/sh # # pipe -- multi-input, multi-output pipelining # # Terry Jones 19/10/87 (tcjones@watdragon) # #------------------------------------------------------------------------------- # Department Of Computer Science, University Of Waterloo # Waterloo Ontario Canada N2L 3G1 # #{ihnp4,allegra,decvax,utzoo,utcsri,clyde}!watmath!watdragon!tcjones #tcjones@dragon.waterloo.{cdn,edu} tcjones@WATER.bitnet #tcjones%watdragon@waterloo.csnet [from oz, tcjones@dragon.waterloo.cdn@munnari] #------------------------------------------------------------------------------- # myname=`basename $0` if [ $# -eq 0 ] then while read line do echo $line done exit 1 fi def_shell=/bin/sh if [ -z "$SHELL" ] then echo No '$SHELL' variable set, using $def_shell SHELL=$def_shell fi cum_in=/tmp/${USER}_pipe_$$ if [ -f $cum_in -a ! -w $cum_in ] then echo ${myname}: could not use temporary ${cum_in} - try again. exit 1 fi >$cum_in tty=`tty` #echo rm $cum_in >> rmt IN=1 SOME_IN=0 SOME_OUT=0 ANY_OUT=0 while [ -n "$1" ] do case $1 in -in|-i) IN=1 SOME_IN=0 if [ "$SOME_OUT" = "0" ] then SOME_OUT=1 cat $cum_in fi shift;; -out|-o) IN=0 SOME_OUT=0 ANY_OUT=1 if [ "$SOME_IN" = "0" ] then SOME_IN=1 while read line do echo $line >> $cum_in done fi shift;; *) if [ "$IN" = "1" ] then SOME_IN=1 if [ "$1" = "-" ] then while read line do echo $line >> $cum_in done else echo $1 | $SHELL >> $cum_in fi else SOME_OUT=1 if [ "$1" = "-" ] then cat $cum_in else eval "cat $cum_in" "$1" fi fi shift;; esac done if [ "$SOME_OUT" = "0" -o "$ANY_OUT" = "0" ] then cat $cum_in fi /bin/rm -f $cum_in @//E*O*F pipe// chmod u=rwx,g=,o= pipe echo Inspecting for damage in transit... temp=/tmp/shar$$; dtemp=/tmp/.shar$$ trap "rm -f $temp $dtemp; exit" 0 1 2 3 15 cat > $temp <<\!!! 141 848 5157 pipe_doc 113 228 2331 pipe 254 1076 7488 total !!! wc pipe_doc pipe | sed 's=[^ ]*/==' | diff -b $temp - >$dtemp if [ -s $dtemp ] then echo "Ouch [diff of wc output]:" ; cat $dtemp else echo "No problems found." eadeade