Path: utzoo!mnetor!tmsoft!torsqnt!news-server.csri.toronto.edu!cs.utexas.edu!wuarchive!zaphod.mps.ohio-state.edu!rpi!rodney
From: rodney@dali.ipl.rpi.edu (Rodney Peck II)
Newsgroups: comp.lang.perl
Subject: archfs -- a virtual file system based on comp.archives data
Message-ID: <0B&&-S^@rpi.edu>
Date: 8 Feb 91 03:12:52 GMT
References: <121642@uunet.UU.NET> <1991Feb07.210023.8308@convex.com> <121661@uunet.UU.NET>
Organization: Image Processing Lab, Rensselaer Polytechnic Institute, Troy NY
Lines: 412
Nntp-Posting-Host: dali.ipl.rpi.edu


  I've been playing with perl some more and I came across an idea for
something that I think is pretty neat.

  Basically, it takes the data from the headers of the messages in 
comp.archives and generates a virtual filesystem.  It's sort of like
restore's interactive mode where you can cd and ls the directories.  At
the leaf nodes, are files which are named after the date they were posted
to the net.

  There are a couple of things that you can do to these files.  The 'article'
command will take that file and fetch the original message from the
usenet spool and give it to more so you can read it.
  The other thing you can do is give the 'ftp' command.  This will try to
read the header and find the hostname, pathname, etc and execute an ftp
command to get the file and put it in your /tmp directory.  It doesn't quite
work yet because I can't get interaction with FTP down properly.  I'm
planning to use Randal's expect like perl code to handle this.  Ironically,
I lost the pointer to it so I have to go poke around until I find it or
someone mails me a message telling me where I can find it.

  Anyway, go ahead and try this out.  Just unshar it, and change the
first bit to point to your spool directory (mine's /usenet/spool).

  the real question is: who's brave enough to take this idea and expand
it into an actual virtual filesystem that the regular command line can
work on?  It would be really nice to be able to say something like
"cp /ftp/hackers/words/file ~/jargonfile.txt"

Let me know about any good changes or additions or perl code suggestions.
I'm still new to this perl stuff.

Rodney

--------------cut here---------------------------------------------
#! /bin/sh
# This is a shell archive.  Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file".  To overwrite existing
# files, type "sh file -c".  You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g..  If this archive is complete, you
# will see the following message at the end:
#		"End of shell archive."
# Contents:  archfs
# Wrapped by rodney@dali.ipl.rpi.edu on Thu Feb  7 21:57:22 1991
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f 'archfs' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'archfs'\"
else
echo shar: Extracting \"'archfs'\" \(8053 characters\)
sed "s/^X//" >'archfs' <<'END_OF_FILE'
X#!/usr/local/bin/perl
X'di';
X'ig00';
X#
X# $Header: /h1/rodney/bin/RCS/archfs,v 1.4 91/02/07 21:56:00 rodney RelNet $
X#
X# $Log:	archfs,v $
X# Revision 1.4  91/02/07  21:56:00  rodney
X# released to the net
X# 
X# Revision 1.3  91/02/07  21:51:30  rodney
X# before adding the expect part
X# 
X# Revision 1.2  91/02/07  21:22:25  rodney
X# added usenet reading directly
X# 
X# Revision 1.1  91/02/07  20:46:46  rodney
X# Initial revision
X# 
X
X# arcfs looks at the archive names from the usenet articles and makes 
X# a mythical filesystem that you can cd and ls sort of like restore -i
X# a special command 'article' lets you get the article from the news
X# spool and look at it with more
X
X$archive = '/usenet/spool/comp/archives';
X$parse++;
X#
X$inodenum = 0;
X@dir[0] = "/:-1";
X
Xif ($parse) {
X  opendir(DIR,$archive);
X  @articles = grep (!/^\./,readdir(DIR));
X  closedir(DIR);
X
X  $count = $#articles+1;
X  print "Getting header info from ",$count," articles.\n";
X  $|++;
X  for (@articles) {
X    $file = $archive . '/' . $_;
X    print --$count,".. ";
X    $head = 0;
X    $base = '/'.$_;
X    open (ARCHIVE, $file) || die "$0: can't open $file: $!\n";
X    while (<ARCHIVE>)
X    {
X      last if (/^$/ && $head++);  # end of second header no archive name entry.
X      next unless /^Archive-name: (.*)$/;
X      &create($1,$base);
X      last;  # that's all we needed
X    }
X    close (ARCHIVE);
X  }
X  $|--;
X}
Xelse
X{
X  open (ARCHIVE,'processed') || die "$0: couldn't open processed:$!\n";
X  while (<ARCHIVE>)
X  {
X    @dir[$inodenum++] = $_;
X  }
X  close (ARCHIVE);
X}
X
X$dot = 0;
X
Xprint "Comp.archives filesystem ready.\n%";
Xwhile (chop($_ = <STDIN>))
X{
X  last if /^exit/;
X  &save if /^save/;
X  &ls if /^ls/;
X  &cd($1) if /^cd (.*)/;
X  &article($1) if /^article *(.*)/;
X  &ftpget($1) if /^ftp *(.*)/;
X  print "%";
X}
X
X
X# return the data field of a file
Xsub getdata {
Xlocal ($art) = @_;
Xlocal ($node) = &getdir($art);
X  if ($node == -1) 
X  {
X    print "file not found.\n";
X    return;
X  }
X  local ($name,$dotdot,$data) = @dir[$node] =~ split(/:/);
X  unless ($data =~ m+^/.*$+)
X  {
X    print "$name is not a file.\n";
X    return;
X  }
X  $data;
X}
X
X# parse the archive header and try to ftp the file to the tmp
X# directory.
Xsub ftpget {
X  local ($dir) = @_;
X  local ($data) = &getdata($dir);
X  if ($data eq '')
X  {
X    print "can't ftp.\n";
X    return;
X  }
X  local ($head) = 0;
X  open (ART,$archive.$data) || die "couldn't open $data: $0";
X  local($archiveline);
X  while (<ART>)
X  {
X    last if /^$/ && $head++; # end of header  
X    if (/^Archive: (.*)$/)
X    {
X      $archiveline=$1;
X      last;
X    }
X  }
X  close(ART);
X  $_ = $archiveline;
X  print $_,"\n";
X  unless (/(.*):(.*) \[(.*)\]/)
X  {
X    print "No readable information in the header.  Sorry.\n";
X    return;
X  }
X  local ($host) = $1;
X  local ($path) = $2; 
X  local ($addr) = $3;
X
X  $path =~ m+.*/(.*)+;
X  local ($file) = $1;
X  
X  open (FTP,"|ftp -n $addr");
X  print (FTP,"user anonymous\n");
X  print (FTP,"password `hostname`");
X  print (FTP,"binary\n");
X  print (FTP,"get $path /tmp/$file\n");
X  print (FTP,"bye\n");
X  close (FTP);
X  print "The file is in /tmp/$file\n";
X}
X
X# fetch the article for this node and give it to more
Xsub article {
X  local ($dir) = @_;
X  local($data) = &getdata($dir);
X  system ("more ${archive}$data") if ($data ne '');
X}
X
X# return the name of the inode
Xsub name {
Xlocal ($node) = @_;
Xlocal ($nodename, @foo);
X ($nodename,@foo) = @dir[$_] =~ split(/:/);
X $nodename;
X}
X
X# get inode for file in this dir
Xsub getdir {
Xlocal ($dir) = @_;
X# break up this dir's inode list
X    local($nodes) = @dir[$dot];
X    local(@nodes) = $nodes =~ split(/:/);
X
X# special case for '..'
Xif ($dir eq '..') {
X  return @nodes[1];
X}
X
X# special case for ''
Xif ($dir eq '') {
X  return @nodes[2] if $#nodes == 2;
X  print "There is more than one file here, please specifiy.\n";
X}
X  
X# look up the dir in the current dir's inodes
X    local(@ls) = @nodes[2..$#nodes]; 
X  for (@ls) {
X    return $_ if (&name($_) eq $dir);
X  }
X  return -1;  # file not found
X}
X
X# cd to the directory
Xsub cd {
Xlocal ($dir) = @_;
X
Xif ($dir =~ m+(.*)/(.*)+)
X{ # it's a pathname
X  local ($first) = $1;
X  local ($rest) = $2;
X  if ($first eq '') 
X  { # absolute pathname
X    $dot = 0;
X  } else
X  { # relative pathname
X    &cd($first);
X  }
X  &cd($rest);
X  return;
X}
Xlocal($foo) = &getdir ($dir);
Xif ($foo == -1) {
Xprint "$dir not found\n";
X} else { $dot = $foo;}
X}
X# list the contents of the current directory
Xsub ls
X{
X    local($nodes) = @dir[$dot];
X    local(@nodes) = $nodes =~ split(/:/);
X    local(@ls) = @nodes[2..$#nodes]; 
X    local($max) = 0;
X    local($len);
X    local(@files);
X
X    for (@ls)
X    {
X      ($nodename,@foo) = @dir[$_] =~ split(/:/);
X      unshift (@files,$nodename);
X      $len = length $nodename;
X      $max = $len if ($len > $max);
X    }
X
X    @files = sort @files;
X    local ($cols) = int(80 / ($max + 2) - 1 + .5);
X    local ($line, $c);
X    local ($rows) = int(($#ls+1)/$cols + .5);
X    $rows++ if ($rows * $cols < $#ls + 1);
X    for ($line=0;$line < $rows; $line++)
X    {
X      for ($c=0; $c < ($cols); $c++)
X      {
X        $nodename = @files[$line + $rows * $c];
X        print $nodename, ' ' x ($max + 2 - length $nodename);
X      }
X      print "\n";
X    }
X}
X
X# save the directory tree to a 'processed'
Xsub save {
X  open (SAVE,'>processed') || die "$0: couldn't open processed:$!\n";
X  for (@dir) {
X    print SAVE $_,"\n";
X  }
X  close (SAVE);
X  print "wrote $#dir nodes.\n";
X}
X
X
X# returns the next inode
Xsub newnode {
X  ++$inodenum;
X}
X
X# create a file, making the directory tree as needed
X# call as '&create(pathname,filedata)'
Xsub create {
Xlocal ($path, $data, $dot) = @_;  # $dot is an internal var which holds the
X			   # current inode -- it's zero from the top.
X  $dot = 0 if ($dot == 0);
X  if ($path =~ m-(\w+)/(.*)-)
X  { # it has directories to descend
X    local ($dirname) = $1;
X    local ($restpath) = $2;
X    local ($in);
X    @nodes= @dir[$dot] =~ split(/:/);
X    for (@nodes[2..$#nodes])
X    {
X      ($nodename,@foo) = @dir[$_] =~ split(/:/);
X      ($in = $_, last) if ($nodename eq $dirname);
X    }
X    if ($in == 0) 
X    {
X      $in = &newnode;
X      @dir[$in] = "$dirname:$dot";  #make new inode
X      @dir[$dot] .= ":$in";         #append to cur dir
X    }
X    &create($restpath,$data,$in);
X  }
X    else
X  { # it's a file
X    $in = &newnode;
X    @dir[$in] = "$path:$dot:$data";     #make new inode
X    @dir[$dot] .= ":$in";               #append to cur dir
X  }
X}
X    
X###############################################################
X
X    # These next few lines are legal in both Perl and nroff.
X
X.00;                       # finish .ig
X 
X'di           \" finish diversion--previous line must be blank
X.nr nl 0-1    \" fake up transition to first page again
X.nr % 0         \" start at page 1
X'; __END__ ##### From here on it's a standard manual page #####
X
X.TH ARCHFS 1 "February 7, 1991"
X.AT 3
X.SH NAME
Xarchfs \- browse the comp.archives filesystem
X.SH SYNOPSIS
X.B archfs
X.SH DESCRIPTION
X.I Archfs
Xtakes the articles in your usenet spool directory for comp.archives and
Xcreates a sort of filesystem from the archive name headers provided by
XEd.
X
XYou can cd and ls this filesystem like it was a normal one.  In this way
Xit is modeled after the restore -i command.
X
XIn addition, you can use the
X.B article
Xcommand to apply the pager to the article from the spool.  Presumably,
Xyou can then read the message and ftp it if you like.
X
XThe
X.B save
Xcommand writes the contents of the database into the file called 
Xprocessed in the current directory.  You can save a lot of time in
Xparsing if you use this once after reading the spool.  Unset the $parse
Xvariable to have it read the processed file instead of looking through
Xcomp.archives.
X
XFinally, there is the start of the code to handle automatic ftp fetching
Xof files.  It does everything up to where it has to talk to ftp itself.
XFor that, I need Randal's expect in perl thing and I've misplaced my 
Xpointer to it.
X
X.SH ENVIRONMENT
XNo environment variables are used.
X.SH FILES
XNone.
X.SH AUTHOR
XRodney Peck II
X.SH "SEE ALSO"
Xcomp.archives
X.SH DIAGNOSTICS
X
X.SH BUGS
X
END_OF_FILE
if test 8053 -ne `wc -c <'archfs'`; then
    echo shar: \"'archfs'\" unpacked with wrong size!
fi
chmod +x 'archfs'
# end of 'archfs'
fi
echo shar: End of shell archive.
exit 0


-- 
Rodney