Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!usc!elroy.jpl.nasa.gov!ames!uhccux!munnari.oz.au!murdu!ucsvc.ucs.unimelb.edu.au!u3364521
From: U3364521@ucsvc.ucs.unimelb.edu.au (Lou Cavallo)
Newsgroups: comp.sys.amiga.tech
Subject: Virtual Disk (Was Re: Files larger than available memory.)
Message-ID: <1124@ucsvc.ucs.unimelb.edu.au>
Date: 6 Oct 90 18:17:35 GMT
References: <924@ucsvc.ucs.unimelb.edu.au> <1990Sep23.174736.16118@lavaca.uh.edu> <1088@ucsvc.ucs.unimelb.edu.au> <409@tlvx.UUCP>
Organization: I.A.E.S.R., Melbourne University
Lines: 110

G'day,

{Please forgive the length of this article. I hope you find it worthwhile.}

The results of this discussion thread have been as great as I was hoping they
would be. Thanks everyone. Hereafter follows a summary and a new idea I think.

Summary:

The consensus (if there is one) seems to be that a swapping library should be
put together that allows new applications to use routines to implement memory
paging for data files. {Dave Haynie's do the work once principle.}

{I support this idea also. Software may need a set of tools rather than a one
size fits all s/w VM solution...analogously to the different alloc routines?}

Mike Meyer and Matt Dillon have considered the case of an editor and the type
of criteria suitable for VM for file accessing.  {Others have suggested their
own ideas here too but I haven't kept track of it all, sorry.}

Some have mentioned Virtual for Mac's (Dave Haynie pointed to a reference for
I believe a way for Mac OS to allocate VM for applications. Si Dave?) and the
VM techniques used on MS-DOS systems. {Virtual is non MacOS VM s/w for Macs.}

A swapping library technique could not offer memory paging for programs that
were not compiled to use it. {Several suggestors here.}

And my original suggestion. Can we find a way to allow (in the interim period
until OS support for VM arrives) applications access files larger than alloc-
atable memory? {Others are suggesting/supporting this limitation also and I'm
again sorry I haven't tracked this as well.}

{There were other suggestions e.g. using more physical memory but I think the
above covers most of them. I offer my apologies if I left your idea out.}

My idea:

A virtual memory fixed disk.

This is not like the VD0: (by ASDG no?) which is a RAM disk that grows as any
data is put in it but only up to the available amount of RAM.

A virtual memory fixed disk (VFD:) would be a file structured RAM disk device 
like RAD: that would set up on it a set of memory swapping buffers for files.
The VFD could be mounted to (say) a 100 Mbyte virtual allocation but only use
the user selected amount of RAM for the in memory buffers.

Naturally real disk storage media sufficient to store the whole of a virtual
file would be required. A 100 Meg virtual file is not going to fit onto your
floppy eh? :-)

Generalisations to recoverable and / or dynamic virtual disk devices are left
as exercises for the reader. :-)

{Note that I'm addressing the last of the summary points here. I'll leave the
question of s/w VM to others but I will note there has already been a (doozy)
of discussion on it in this newsgroup.}

This suggestion is specifically to deal with files larger than (allocatable)
memory. It is not a way to give VM for programs for that need allocations for
RAM memory objects larger than that available (I think).

Discussion:

Pros:

All programs and not just those linked with a swapping.library or vm.library
could access files larger than allocatable RAM.

If a user's or programmer's problem requiring VM can be solved using a file
then VFD can provide a crude VM facility.

No changes to the OS should be necessary for this to be implemented assuming
a swapping.library set of routines that a VFD: device handler could be built
with.

This idea (I feel IMHO) solves the problem in a style suiting the Amiga. The
device handler could possibly be written for use with any type of filesystem
or trackdisk level interface. {I defer here, I don't know Amiga internals.}

Cons:

I don't know how to do it. :-)  {Discussions as to the actual structure of a
VFD are sought here.}

Would the overhead of maintaining the virtual files in VFD be too much for a
good/acceptable level of performance?

Files put onto a VFD would be in a virtual memory file system but should act
as normal files do. Would there be any problems with features of the Amiga's
file system (say links, or deleting files that are not unlocked) with respe-
ct to the fact that its really implemented in RAM? (I don't think so but...)

VFD: files could be corrupted by errant tasks/processes in memory via trash-
ing of the swapping buffers by errant memory writes.

Large excutables (larger than available RAM) put on a VFD device should still
fail to load into memory as the OS will want to load all of an executable. {I
simplify here, program overlays could be used but assume the overlays are all
too large also.} Actually I'm not sure I'm right here. Perhaps it would work?

yours truly,
Lou Cavallo.

PS: If I've made any obvious points of expository inconsistency it's probably
because I tried to keep each point to a brief minimum.

So what do you think ?  Is this a good level to deal with the solution to the
problem or is it all wet and would s/w VM be a simpler/better/more_useful way
than dealing with the one problem of files that are too large?