Path: utzoo!utgpu!news-server.csri.toronto.edu!smoke.cs.toronto.edu!moraes
Newsgroups: comp.archives.admin
From: moraes@cs.toronto.edu (Mark Moraes)
Subject: Re: New version of my ftpd is now available
Message-ID: <91Jun17.215218edt.1568@smoke.cs.toronto.edu>
Organization: Department of Computer Science, University of Toronto
References: <3661@sirius.ucs.adelaide.edu.au> <1991Jun16.092611.15695@newshost.anu.edu.au>
Date: 18 Jun 91 01:52:34 GMT
Lines: 78

To stave off the mail messages:  The original version of the scripts
Albert refers to can be found on ftp.cs.toronto.edu as
pub/ftptools.shar.Z.  There's also a modified ftp client there -- it
creates the intervening directories (like mkdir -p) locally if
necessary.  (I used them for a long time with a vanilla ftp client,
but got tired of the nuisance) If you make fixes or improvements to
them, I'd appreciate a copy.

>The CHANGE file in ftpd.5.60.tar.Z mentions that with the new version of
>ftpd for BSD systems, ftp can no longer directly run ls.

All anon. ftp sites providing an regularly updated ls-lR.Z in ~ftp
would be a better solution anyway -- some sites may prefer it to
having other people grind their disks with a periodic "ls -lR".  (If a
hundred people started running archie servers, I wouldn't like to
think of the effect on archive site disks)

And then there's the problems when your "ls -lR" scans someone's /mnt
or /afs directory with many other systems mounted there.

Miscellaneous points from the last time we had this discussion on some
archive related mailing list (archie-people?):

- If the format isn't relatively standard, it isn't too useful.  In
that sense, "ls -lR" output is nice and standard.

- If the location of the file isn't standard, it's annoying.  Some
sites put it in /, some in /pub, some in /dist, etc.
	if ftptest -f ftp.wherever:pub/ls-lR.Z

- Even "ls -lR" output is somewhat redundant - the owner, even the
modes of the file aren't very interesting (unless the file is
unreadable.)  And SysV ls has to be given -g or it sticks the group
info in there.

- It's best to run the ls -lR as the ftp user, or nobody, so the files
that are listed are ones that the user can list automatically.

- the date in "ls" output is somewhat annoying from an automatic
archive tracking package, since it goes backward every six months as
the hh:mm go to zero.

- Columns are a real pain for an automatic listing script -- I have
encountered an ftp site that had a columnated ls -R output; useful for
humans, not quite so useful for scripts.

One possibility is for (Unix) sites to run
	ls -lR | ftpfilter -l | compress > FILES.Z

The format that my "ftpfilter -l" produces from ls -lR is:

name bytes YYYYMMDDhhmm

Directory names have a trailing /.  bytes is sort of meaningless for a
directory.  And please don't read me back my news.software.b postings
about all-numeric dates :-)

eg.

README 1191 199012110000
ailist/ 3072 199001190000
bin/ 512 198811200000
ca-domain/ 3584 199104180410
comp.archives/ 1024 199106032020
ailist/V. 30267 199103240154
ailist/V7.10.Z 9242 198806020000
ailist/V7.11.Z 12851 198806020000
ailist/V7.12.Z 8566 198806020000

This form is grep'able, which is a big win over ls-lR, and it
compresses well -- 70% is typical, which compensates for the expansion
caused by pathnames.  For example the ls-lR.Z of ftp.cs.toronto.edu's
files is 104K.  The compressed ftpfilter output is 106K.  uunet's
ls-lR.Z is 376K, the compressed ftpfilter output is 356K.  (even better
compression can be achieved by prefix compression before LZW, but it
isn't worth the cost of the extra non-standard program).

	Mark (don't call me Tony!) Moraes