Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!zaphod.mps.ohio-state.edu!ub!uhura.cc.rochester.edu!rochester!udel!ee.udel.edu
From: new@ee.udel.edu (Darren New)
Newsgroups: comp.sys.amiga.tech
Subject: Re: Files larger than available memory.
Message-ID: <32605@nigel.ee.udel.edu>
Date: 6 Oct 90 00:41:41 GMT
References: <1990Sep27.124036.9866@msuinfo.cl.msu.edu> <1990Sep27.154312.21380@msuinfo.cl.msu.edu>
Sender: usenet@ee.udel.edu
Organization: University of Delaware
Lines: 53
Nntp-Posting-Host: estelle.ee.udel.edu

I have a library that I use when I want to work with files larger than
memory.  It has several features that I find indispensible:

1) The virtual memory regions are randomly addressable. That is, there
is none of this "read an area into your own memory and then use it from
there" baloney.  Having the facility to directly access memory makes
many operations much easier. Without it, you might as well use files.

2) The VM is associated with a particular file. Hence, one can "open"
the file, access it like memory, and then "close" the file with the
assurance that the info in the file will still be there next time you
open it.  Combined with (1), this allows applications to have "global
variables" whose lifetime exceeds that of the program.  There is no
need to write routines to file-out and file-in the memory structures
while correcting pointers and etc.

3) There are routines for efficient malloc/realloc/free-style calls,
and eventually I will add GC and such. Generally not difficult unless
you have to reinvent them every time you need them.

4) There are calls to dump the VM in a readable format for ease of
debugging.

5) I have the source, and it's fairly portable :-)

There are three features it ought to have that it doesn't:

1) You have to use special macros to access the VM, so you can't (for
example) map a structure onto the VM region and use it there.  So far,
my structures have been simple enough that I could get away with L =
GM(offset) to get a long and C = GM(offset,byteoffs) to get a character
with similar routines for storing values. It would be nice to be able
to use standard unary * notation; however, I don't think this could be
done on a system without an MMU.

2) You can't have more than one VM file open at once.  I may rewrite
mine to allow this, as I am currently trying to think of how to do my
latest project without this and it would be much easier with this.

3) You can't have more than one process accessing the same file at the
same time consistently. So far, this is not a problem for me because
all programs using the VM are servers to other programs (via ARexx,
mostly).  However, I can see where some programs would want this.

All in all, some subset of the mmap() functionality is what I find
useful.  Possibly just "Open a VM file and tell me where you put it"
would be enough, given that the other exec memory allocators would work
on that area.  Again, this won't work on a nonMMU system.
         -- Darren
-- 
--- Darren New --- Grad Student --- CIS --- Univ. of Delaware ---
----- Network Protocols, Graphics, Programming Languages, 
      Formal Description Techniques (esp. Estelle), Coffee -----