Xref: utzoo comp.dcom.lans:1752 comp.protocols.tcp-ip:4458
Path: utzoo!utgpu!water!watmath!clyde!att!osu-cis!tut.cis.ohio-state.edu!mailrus!cornell!uw-beaver!uw-june!sung
From: sung@june.cs.washington.edu (Sung Kwon Chung)
Newsgroups: comp.dcom.lans,comp.protocols.tcp-ip
Subject: Re: RUNNING FTP IN BATCH MODE
Summary: shell scripts for batch mode ftp, need Unix (-like) shell
Keywords: ftp, batch, shell
Message-ID: <5576@june.cs.washington.edu>
Date: 28 Aug 88 06:48:11 GMT
References: <309@versatc.UUCP>
Reply-To: sung@uw-june.UUCP (Sung Kwon Chung)
Organization: U of Washington, Computer Science, Seattle
Lines: 148

Here are two quick solutions to do batch-mode ftp.  They are written in
shell script, that means they are of no use for some of you who don't have
the shell, sorry.

The first one (ftp1.sh) is for simple ftp'ing.  It works fine for small number
of files.  The limit depends on the command buffer size of `mget' and some 
other factors unknow to me :-)  Is is used like,
	% ftp1.sh host pub a b c d e f g
to get file a, b, ..., g on host:pub.

The second one (doftp) is a bit more complicated, but versatile (and
extensible if you want). doftp uses other two shell scripts (ftp.sh and 
waitfor). It requires a file which contains the names of the files to be
ftp'ed.  Each line in the file contains one file name.  If `files.list'
contains wanted file names on host:pub,
	% doftp host pub files.list
will transfer all of them (of course, if everything goes right).
I used it to bring over 70 odd (irregularly selected) files from
an archive site.

To use them, extract the scripts to their names and do "chmod +x" them.
Please see the notes at the end of this message before trying them.
========================= ftp1.sh =========================
#!/bin/sh
#
# simple ftp 
# Usage: ftp1.sh host direcrory file1 file2 file3 .....
#
host=$1; directory=$2; shift; shift;
ftp -n -i $host << ___EOF___
user anonymous guest
cd $directory
mget $*
bye
___EOF___
exit $?
------------------------------------------------------------


The following three files are for the `doftp'.
========================= doftp =========================
#!/bin/sh
# Usage: doftp host directory namelistfile
# needs ftp.sh and waitfor
# Set up the pipes and temp files, and invoke ftp and its command feeder
#     ftp-out$$ : conversation during the job, feed-back to ftp.sh
#     ftp-log$$ : just for logging the commands
#	sung@cs.washinton.edu, 8/22/88
if test "$#" != "3"
then
    echo "doftp host directory namelistfile" 1>&2
    exit 1
fi

oldmask=`umask`		#
umask 077 		# if we need to make the conversation unreadable 
cat >ftp-out$$ </dev/null
cat >ftp-log$$ </dev/null
umask $oldmask

( ftp.sh  $2 $3  <ftp-out$$) |
                     tee ftp-log$$ | 
                                    ( ftp -v -n $1  >ftp-out$$ 2>&1 )
------------------------------------------------------------
========================= ftp.sh =========================
#!/bin/sh
# ftp.sh: ftp command feeder
#
# Uage: (sleep 1; ftp.sh dir namelistfile <ftp-out) |
#					 (ftp -v -n host-name >ftp-out) 
# See `doftp' and `waitfor'
#	sung@cs.washington.edu, 8/22/88

if test "$#" != "2" 
then
	echo " Uage: (sleep 1; ftp.sh dir namelistfile <ftp-out) | (ftp -v -n host-name >ftp-out)" 1>&2
	exit 2
fi

echo "user anonymous guest"		# login
result=`waitfor "230"`			# okay?
if test "$result" = "ERROR"
 then exit 1
fi

echo "cd $1"
result=`waitfor "250"`			# 250 CWD successful
if test "$result" = "ERROR"
 then exit 1
fi

# other FTP commands here, if needed

for f in `cat $2`		# main loop
do
   destName=$f	# file name change may be used here.  a:b.c --> a-b.c
   if test ! -f $destName	# get it, only when it is not there
   then
      echo "get $f $destName"
      result=`waitfor "226"`	# 226 Transfer complete
      case "$result" in
      "OK") echo "got $f" 1>&2 ;;
      "ERROR") echo "failed getting $f, abort the job" 1>&2;
               exit 1;;
      # other cases may need to be considered here
      esac
   fi
done
exit 0
------------------------------------------------------------
========================= waitfor ==========================
#!/bin/sh
# Usage: result=`waitfor string`
#    wait for given string from std input while monitoring error messages
#    Usually, `string' is an expected message number (e.g., 250).  It
#    detects 5xx error messages.
#	sung@cs.washington.edu, 8/22/88
while true
do
  read aLine
  if  test "$aLine" != ""
  then
    case "$aLine" in
    "$1"*) echo "OK"; exit 0 ;;
    *'timed out'*) echo "ERROR"; exit 1 ;;	# timeed out message
    5*) echo "ERROR"; exit 1 ;;			# 5XX error msg
    *) ;
    esac
  else
   sleep 3
  fi
done
------------------------------------------------------------
The scripts are for anonymous login.  For non-anonymous login, you can
change the `user' command line. (Be sure to set proper file protection
in this case.) Or you can use .netrc file and auto-login feature.  In this
case, '-n' option is in `doftp' should be removed.

There is much room for improvement especially for error message handling,
and retrial in the case of partial fairlure.  So please feel free to
make further extensions or modifications as needed.

Hope this helps some of you.

Sung K. Chung                     |       sung@cs.washington.edu
Dept. of Computer Science, FR-35  | {decvax,ucbvax}!uw-beaver!uw-june!sung
University of Washington          |
Seattle, WA 98195                 |