Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!uunet!isis!chanson
From: chanson@isis.cs.du.edu (Chris Hanson)
Newsgroups: comp.sys.amiga.tech
Subject: Re:   Files larger than available memory.
Summary: Entering _BRAINSTORM_ mode.
Keywords: Software, VM, Software VM, it's late, need more caffeine
Message-ID: <1990Oct13.031948.25857@isis.cs.du.edu>
Date: 13 Oct 90 03:19:48 GMT
References: <lron.1044337@easy.HIAM> <1990Oct7.032409.22928@zorch.SF-Bay.ORG> <lron.1044713@easy.HIAM> <1990Oct9.061755.15112@zorch.SF-Bay.ORG>
Reply-To: chanson@isis.UUCP (Chris Hanson)
Organization: Matrix
Lines: 101


   How about this concept: Suppose you went to edit one of those vaunted and
overwhelming 6 meg files, on your 512k machine. Your editor, (being smart,
clever, and posessing much farvergnugen) realizes the impossibility of the
situation, and calls on virtual.library for help.
 
(Virtual.Library, clad in red-n-blue tights, enters from stage left.)

   So anyway, the editor opens virtual.library, and calls SetupVirtualBlock(),
telling it how big of an area it wants (6 stinkin' meg), and optionally, where
to load that 6 meg of data from. (Could speed things up a bit if the library
was informed of the source of that data, but it isn't important.) Ok, the 
library returns some code stating that the planets ARE in alignment, and
that everything is in fact groovy. The editor opens its screen, and goes to
display the first page of text. Oh. No text. So, the editor calls on the
virtual.library RealizeChunk(), (to make a chunk of 'virtual memory' into
'real memory') passing it a 'size' parameter, (oh, about 80 x 25 or about
2k, more than enough text to fill the screen) a pointer to where our
virtual.library should put the data, and a index of where in the 6m file it
should get the 2k of data. (Probably the beginning.)

   The request is good, and virtual.library deals with it, loading the data 
either from the source file or from its temporary file, and putting it into
the memory buffer that the editor pointed out. Great. Now you start editing.

  As you insert characters (say, adding in one of these little notes-in-
parenthesis things that I use alot), the editor displays the changes
onscreen, while doing a virtual.library InsertChunk(), telling virtual.
library where, how big, and what was inserted. Virtual, being a very clever
beast, just queues these up, until you stop doing InsertChunk()s, and do 
something else. When you, for example, do a couple of page-downs in the
editor, the editor does a RelinquishChunk() on the 2k chunk you were 
working on, and does RealizeChunk() on another area a couple pages down.

   The virtual.library (being clever, et al) notices that you stopped
doing all those ninny one-character inserts, and decided to go work somewhere
else. It Groks its way through its queue of all the InsertChunk()s you did,
and builds a new version of that 2k block of memory that you were working
on, a new version which is, oh, 2.5k after all the additions you made.
It writes the new 2.5k block to the disk in its temporary area, noting that
this 2.5k block now takes the place of the 2k block in the original file. 
Then it goes to Work on the new area of the file that you page-down'ed to.
The RealizeChunk() succeeds, and the editor has another ~2k block to 
work with.

  This time you get really daring, and delete some reference
to IBM-multitasking-under-MSDOG from your file. A big reference, that
amounts to about, 512 bytes. You can delete it as a block cut, or sit there
and hammer away on the backspace key until it is gone. The editor will
do one big DeleteChunk(location, size) or a bunch of little DeleteChunk(
location, 1)s. Either way, you finish erradicating that fiction, and decide
to skip to the end of the file. The editor does RelinquishChunk() on the
area you modified, and RealizeChunk()s the end of the file. (Two notes
here, we might be nice, and allow start-relative, current relative, and 
end relative positioning, like fseek does, or we just might make the editor
keep track of how many bytes are in the file currently. You chose. Also,
the editor might have to do another RealizeChunk() after the cut operation,
in order to repaint the entire screen.)  

  So, when the editor did RelinquishChunk on the chunk that had had some
stuff deleted out of it, the virtual.library notes this too, and comes
up with the new, final representation of that ~2k block, which is now 
around 1.5k. And it writes this too to its temporary area, noting that
this 1.5k chunk now replaces this other 2k area in the old file. And 
then, having cleaned THAT up, goes on to give you the last 2k in the file
to edit.

  Ok, so the file looks good to you, and you SAVE'n'CLOSE it. The editor
does a UpDateVirtualBlock() with the necessary arguments, and then a
CloseVirtualBlock(). When it does UpDate...(), the virtual.library goes
into serious action, and builds a quick list of all of the locations in
the original file where something has been changed or added/deleted. It
then (possibly copies the original file and) opens the original file,
fseeks to the first modification, fwrites out _ITS_ version of the modified
area, writes out all the unchanged data between there and the next
modification (getting the unchanged data from the copy it just made),
writes out _ITS_ version of the next change, etc, etc, etc.

It is even cleaner if you don't add/delete data, and just modify it. Because
RelinquishChunk can then just modify the working copy of the file right
then, rather then waiting for you to quit, and then rewriting the ENTIRE thing.

Basically, it involves just asking for, modifying, and relinquishing
certain blocks of a known file. We could be nice and put in functions
to quickly (or, as quickly as POSSIBLE) search a given block for strings,
etc.
 
   As my summary sez, it is late, I need more caffeine. Please,
1.DH0:> Flame >NIL:
1.DH0:> Constructive_Comments >ME:
 
    Chris - Xenon

r


-- 
#define chanson Christopher_Eric_Hanson || Lord_Xenon || Kelson_Haldane 
I work, but you don't know who I work for. And they don't know I'm here.

::I'm @ chanson@nyx.cs.du.edu