Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!uwm.edu!zaphod.mps.ohio-state.edu!samsung!think!mintaka!ogicse!schaefer
From: schaefer@ogicse.ogc.edu (Barton E. Schaefer)
Newsgroups: comp.mail.mush
Subject: Re: Huge folders
Message-ID: <7088@ogicse.ogc.edu>
Date: 2 Feb 90 22:24:21 GMT
References: <15147@bfmny0.UU.NET>
Reply-To: schaefer@ogicse.UUCP (Barton E. Schaefer)
Organization: Oregon Graduate Institute (formerly OGC), Beaverton, OR
Lines: 78

First, some introductory remarks on folder compression:

In article <5C98ACE2A8@crdos1> davidsen@crdos1.crd.ge.com writes:
} 
}   If this gets added, and I would love to see it, provision should be
} made to provide a compress and uncompress string, which, if defined,
} would be loaded with the parameters and executed. This would allow not
} only compress, but also things like arc, zoo, zip, lzhuf, etc, archives
} of folders. Many of these run on DOS as well as UNIX.
} 
}                            Make it work right

Of course.  As I pointed out in an earlier article, the main difficulty
with compressed folders is loading from a pipe.  If you're willing to
live with two temp files -- uncompress to the first temp and load it into
the "working" temp from there, then recompress back to the original after
update -- it could be implemented easily.  But in that case you might as
well use the "zfolder" cmd and scripts I'm about to post new versions of,
because that's what they do.

}   This should be integrated into the save commands, too. No use allowing
} a compresses folder if you can't add to it. And perhaps an option to not
} recompress until exit, so going thru your mail and sending stuff into
} folders would not thrash folders if multiple things were added.

See my earlier comments on the infeasibility of "save" into a compressed
folder.  I see what you're driving at -- uncompress when the first save
to that folder is issued, then remember that you need to compress again
at exit time -- but I really think my scheme of saving to a secondary
folder that is not kept compressed, and then merging when necessary, is
more efficient both in terms of time (assuming that the most recently
saved messages are the ones that are most frequently needed, accesses
are quicker because you need not uncompress) and in terms of disk space.

In article <48677cdf.20b6d@apollo.HP.COM> ced@apollo.HP.COM (Carl Davidson) writes:
} 
} I, too, have some mail folders that are huge (> 2 Mbytes). The ability
} to compress/decompress folders "on-the-fly" would be nice. Even better
} would be to store mail messages in a hypertext database.

Goodness, not asking for much, are we? :-)

} I also realize that this is a pipe dream, so I would gladly settle for
} auto compress/decompress.

If that other Davids*n's user-specifable packing/unpacking strings get
implemented, you can probably use them to connect folder loading to any
kind of database you like.  It's a little beyond what mush is designed
to be to have that kind of database manager built in.  (Dan is free to
contradict me on this. :-)

In article <15147@bfmny0.UU.NET> tneff@bfmny0.UU.NET (Tom Neff) writes:
} I too have noticed that folders tend to get huge.  When they do, Mush's
} performance gets excruciatingly slow.  Loading them means scanning
} the entire file searching for message starts.  This can take forever!
} 
} So another idea occurs to me - how about INDEXING huge folders?

Various means for implementing this very thing have been under discussion
for some time.  What hasn't been solved is detection of corrupted folders
or index files, when the index appears valid to external checks (like the
modification times) but actually doesn't agree with the folder.  The
algorithms for doing this validation are understood, but implementation
appears to require a complete rewrite of the folder loading code (which
was hard enough to get right in the first place).

In other words, it's on our "some day" list.

} A final optimization for huge folders would be to update the original
} file IN PLACE if the user's changes don't require moving any text
} around, e.g., deleting new messages while leaving old ones untouched.

Hmmm ....
-- 
Bart Schaefer                             "Live and don't learn, that's us."
                                                                   -- Hobbes

schaefer@cse.ogi.edu (used to be cse.ogc.edu)