Path: utzoo!utgpu!news-server.csri.toronto.edu!bonnie.concordia.ca!uunet!wuarchive!udel!ee.udel.edu
From: new@ee.udel.edu (Darren New)
Newsgroups: comp.os.misc
Subject: Re: What constitutes a good OS?
Message-ID: <42128@nigel.ee.udel.edu>
Date: 18 Jan 91 22:12:00 GMT
References: <5331@auspex.auspex.com> <42010@nigel.ee.udel.edu> <5362@auspex.auspex.com>
Sender: usenet@ee.udel.edu
Organization: University of Delaware
Lines: 287
Nntp-Posting-Host: sam.ee.udel.edu

In article <5362@auspex.auspex.com> guy@auspex.auspex.com (Guy Harris) writes:
>>which you can no longer do.
>I can still use "cat" on directories on the local machine, 

Sorry. I forgot to test this on a local disk, and only tried it over nfs,
which told me Phhhtttt!  

>>But if the filesystem supports keys, then your editors probably do also.
>I wouldn't assume that - VMS includes keyed file support, but I don't
>think its editors support keys.

Ok. Again, my mistake.  Every system that had keys built in I've used has
had editors that could edit based on keys.

>(I assume by "editor" you mean "text editor"; 

Yes.  The problem in part is that this conversation appears to have encompased
many different areas in the same thread.  I started out saying "I want keys"
and one reason was "so the editor could use them". Hence, editors (text editors,
that is) got mixed up into the conversation. Then we got mixed up with
directories (I don't remember how) and object-oriented file systems and
lots of other stuff.  I probably am mixing and matching from different
sub-threads. Sorry.

>were plain text files or not, "files you can read with "open()",
>"read()", and "close()", 

I was trying to distinguish "plain text" from "block special" or
"character special."  Clearly, I failed by chosing the wrong word and
by testing only the NFS files.  Never mind.

>VMS's editors do, I think, support line numbers - but those aren't keys.

Well, the difference in my mind is that line numbers of later lines
change when you insert lines before them, whereas keys don't. How do
VMS's line numbers work?

Lets take this as a definition:
line -- a piece of information in a file that is of fixed size once written.
record -- a piece of information in a file that can be replaced by a 
       different-sized piece later.
line-number [of a record] -- the ordinal number of lines or records before 
	this line or record.
key -- an arbitrary identifier associated with a line or record which can
	be used for random access. Sometimes (usually in text editors) these 
	keys are numbers, giving rise to confusion between keys and 
	line numbers.

>>>So, in RSX-11, are keys in "the filesystem" or in "a library"?
>>Both.  Why should I have to choose?
>Because, in an earlier posting, you said:
>	If keys are in the filesystem as opposed to a library,
>	then every program has access to that information automatically.
>which indicates that you do *NOT* consider them equivalent, and
>therefore that keys *cannot* both be in "the filesystems" and in "a
>library".

OK.  As I tried to say later, CORs (which I called a library above)
which are "connected" to the application when the application opens a
file (at open-time) would be considered (by me) to be in a library and
in the filesystem. Routines (like the ISAM libraries which people have
mentioned) are not in the filesystem because not all programs use them
because they have to be present when the program is compiled (link-time
libraries). In this case, I meant "library" as "collection of
routines," which I shall henceforth abbreviate as COR.

>Please be more careful in the way you phrase things, and more consistent
>between postings; it's hard to consider ideas - and I think a lot of
>what you have to say is worth considering - when you have to spend
>cycles interpreting the way they're phrased due to unusual use of
>descriptive phrases, or have to shift mindset from posting to posting
>because in posting A something in "a library" wasn't the same as
>something in "the filesystem" while in posting B they were the same. 

I apologise. Henceforth, I'll try to follow up with mutliple postings
quoting only the relavent bits. Otherwise, I tend to use your terminology
sometimes and my terminology sometimes, and where mine does not match
yours, confusion reigns. This thread is especially bad, because I'm
using words that everybody knows what they mean (library, file system,
kernel) but nobody really agrees.

>>The real choice is between a user-mode library and the kernel, and
>>maybe even that can be resolved.
>
>OK, so now if keys are in "a library", a program may still have access
>to that information automatically (*every program* doesn't, unless
>somehow the library can manage to *completely* hide the management of
>the keys from the program, or every program is written to use the calls
>that manage keys), and having keys in "the filesystem" is no longer
>opposed to having them in "a library"?

Consider this:
Other than the directory management stuff, did switching from the V7
disk layout to the BSD4.x layout "somehow ... manage to *completely*
hide the management of keys [in this case, file names] from the
program"?  Yes, because every program *was* written to use the calls
(open(), ...) that manage the "keys" (in this case, file names).
Programs that *didn't* use the calls (fsck) had to be rewritten. In
this case, I'm drawing an analogy between keys as record identifiers
and keys as file identifiers. In the same way that a filesystem with
keyed files in it can present keyed files to an application that does
not care about keys in such a way as to make keyed files look like
consecutive files, the BSD NFS routines present an IP network to an
application and made it look like a local disk with files on it. An
application compiled without any non-local disks can nevertheless
access non-local disks as collections of files when moved to an
appropriate machine; such a change does not require any special calls.

>>*** The other choice (which is what you are actually asking) is a
>>choice between a link-time library and an open-time library.  If I need
>>to code special calls into my application to handle different formats
>>of files,
>What exactly do you mean by an "open-time library", and how does the
>above have anything to do with the choice between a "link-time library"
>and an "open-time library"?

Ah, here is the real crux. If the library behaviour is determined by
the file you opened, then it is an open-time library (a collection of
routines, or COR). If it is determined by what was compiled into the
program, it is a link-time COR. (I make up the phrase COR to attempt
to reduce ambiguity about "library".)

The choice between NFS routines or the local disk management routines
is made when you open the file. If I want to create a new layout (say,
one which stores an entire collection of files on one file called a
"stuffed file"), I can do it in one of two ways: I can write a third
open-time COR, and all my binaries (cat, vi, cc, ...) can access my
stuffed files, or I can make it a link-time COR which only my own
applications can use.  I would probably wind up writing a program to
"unstuff" files which would have to be run before I could compile
programs. UNIX already does this to some extent with tar and SCCS and
so on. Imagine that SCCS was written like NFS instead of like tar.
What would you have?  Maybe something similar to VMS's file
generations? Except that with SCCS, you need to check in and check out,
whereas with VMS every program knows how to handle generational files.

>Maybe
>when *reading* a file, you can have an interface to get only the data,
>not the keys, from a file; that doesn't work when writing out a new
>file, unless your library can magically figure out what key should be
>assigned to each unit of the file.

True. However, if you are writing a file from scratch and the standard
text editor normally numbers a file from 100 in steps of 10, then maybe
the interface COR could "magically" chose the same defaults.

In reality, I've used systems where you could not open a keyed file
for writing in consecutive mode, for exactly the reason you stated above.
In practice, this wasn't a problem.

>One way to solve that *without* teaching "cp" about ".dbm" files is to
>have it check for all-zero blocks and write them out as holes.

Certainly. And you have to do this also with tar, cpio, ....  What about
all those other programs you don't have source to?

>>One of my complaints with the UNIX file system is that none of the tools
>>that make UNIX useful can handle any of the complexly-structured files
>>I need to build. If the mechanism for complex files was in place from
>>the start, then all the tools would handle them.
>I don't believe that - you'll have to prove it.  As indicated, that
>requires that "all the tools" know about any stuff that can't be hidden
>in the library interface.

Look at the macintosh.  Every program I know of can handle the resource
fork. Sometimes it isn't visible to the user, but if you open any font,
you have just used a complexly-structured file. Even the code segments
are in the resource fork, just like VM paging is in UNIX. Saying that
tools on the Mac don't all use "the mechanism for complex files" is like
saying that not all programs under UNIX use virtual memory.

I think I've sufficiently "proven" my point by an example of an OS
where the complex file systems were there from the start.

>>Even a simple mechanism whereby a portion of the middle of a file could
>>be replaced by a different size portion (records), and preferable able
>>to be named and retrieved in order (keys), would make many of my programs
>>much easier.
>
>Do you need that ability for *all* files, or would an ISAM file type -
>such as you'd have were all UNIXes to come standard with ISAM libraries
>- be sufficient?

It would also be *nice* if reading an ISAM file with plain old "open()" would
return the keyed records as newline-separated bytestreams in order of
the keys.

>the issue of the RSX libraries
>*perhaps* permitting the *same* calls being used to read, and perhaps
>write, both formats.

That is what I want.

>(In other words, can you use PIP to copy an ISAM file, and have the
>copied file usable as an ISAM file?)

I would hope so.  Maybe I have not used any *really bad* implementations
of a complex file system.

>>A library. I would be possible and easier to write tools that did not
>>use the library
>It's *possible* to do that under RMS; just do your own QIOs.  It may be
>a *pain* to do so, but I'm not sure that makes a difference.

I think it would.  I think most applications that didn't need the extra
complexity would not include it. Under RMS, as you say, you need extra
complexity in the code to make it *not* work.

>*ONCE AGAIN*, and please *ANSWER* this question:
>How are arbitrary programs that *generate* text files to assign keys to
>them, or is it the case that there's no *need* for them to generate
>keyed files?

I would think that if the human user is not assigning keys (like
compiler error message listings) then unkeyed files would be OK.
This assumes there would be a utility to put the "standard" keys for
the text editor onto the file. Not every file needs keys, even if they
come with the OS.

>That depends on whether the filesystem's notion of what a "paragraph" is
>matches what all those programs want.  

True. I was making an analogy and didn't expect you to take me literally.

>>>Again, the question of "what is 'the filesystem'?" comes to mind.
>>In my mind, the filesystem is the interface between applications and
>>the kernel for purposes of doing input and output operations.
>OK, what does "the kernel" mean here?

Umm... the part that isn't an application?  I'm not sure where
user open-time CORs come into this. Basically, I don't knwo how
to define this cleanly and will attempt to avoid "kernal" in the
future.

>Is it "the privileged supervisor"?

The privledged supervisor is certainly wholely part of the kernel. I
was thinking something like the things that are accessed via calls in
section two of the UNIX manual.  Not all things accessed through the
kernel are necessarily privledged or supervisor-resident, nor are they
filesystem-related (e.g., signal()).

>If so, *please* make up your mind - earlier *in this very posting to
>which I'm responding* you indicated that code in a user-mode library
>could be part of the file system.  Now you're saying it has to be in "the
>kernel".  You can't have it both ways....

In the case where user-mode open-time CORs get attached to the
application when the application opens the file, the CORs are not part
of the privledged supervisor but are part of the filesystem. In this
case, the kernel calls the usermode CORs, which define in part the
interface between the application and the stored data (the
"filesystem"), but clearly the CORs are not part of the privledged
supervisor. Here, parts of the filesystem lie outside the kernel but
are presumedly (for security) accessible only through kernel calls.

Analogise with a UNIX system where NFS is a user-mode process and attempts
to talk to a NFS file were intercepted by the NFS deamon. There, NFS would
be part of the filesystem (as it defines the interface between the application
and its data) but not part of the privledged supervisor. If the NFS deamons
provide services that other parts of the filesystem don't (say, something
that reports via IOCTL the load on the system holding the file) then I
would say that (i.e., the new call) is part of the filesystem, just as
/dev/pty became part of the filesystem between V7 and BSD.

>OK, fine, so as of this paragraph, the filesystem is *NOT* the interface
>between the application and the kernel, it's the interface that typical
>applications use to get at files *regardless* of whether it runs in
>privileged mode or not.

OK, you are right.  If the filesystem is entirely outside the kernel,
then the above is correct, as it is in AmigaDOS.  However, on all
multiuser machines I have ever seen, there is a desire to limit access
to the filesystem, and hence access to files has always been mediated
by a privledged chunk of code that cannot normally be bypassed, which I
called the kernel above.  This was the assumption that I think caused
some of the confusion. I was indeed mixing bits of different things. I
would be quite interested in studying any system where the files are
secure but not guarded by a privledged bit of code enforced in
hardware.

		-- Darren
-- 
--- Darren New --- Grad Student --- CIS --- Univ. of Delaware ---
----- Network Protocols, Graphics, Programming Languages, 
      Formal Description Techniques (esp. Estelle), Coffee, Amigas -----
              =+=+=+ Let GROPE be an N-tuple where ... +=+=+=