Path: utzoo!censor!geac!torsqnt!hybrid!scifi!bywater!uunet!auspex!guy
From: guy@auspex.auspex.com (Guy Harris)
Newsgroups: comp.os.misc
Subject: Re: What constitutes a good OS?
Message-ID: <5392@auspex.auspex.com>
Date: 20 Jan 91 21:43:14 GMT
References: <42010@nigel.ee.udel.edu> <5362@auspex.auspex.com> <42128@nigel.ee.udel.edu>
Organization: Auspex Systems, Santa Clara
Lines: 237

>Sorry. I forgot to test this on a local disk, and only tried it over nfs,
>which told me Phhhtttt!  

Yup.  Many NFS implementations forbid using "read()" to read directories
over the wire; this was done to catch naive programs that still thought
"read()" would give you what you wanted (NFS implementations would
mostly give you the *raw bits* of the directory, if they gave you
anything at all, rather than the directory entries in some "canonical"
form understandable by the system reading them; an alternative might
have been to have an NFS read operation on a directory act like a "read
directory" operation).

This is only indirectly connected with the difference between the V7 and
BSD file systems.  NFS is *not* a BSD-ism; SunOS was the first system to
have it - BSD, as in "a system you get from Berkeley", didn't have it
until 4.3-reno - and systems with V7/S5 file systems, BSD file systems,
DOS file systems, VMS file systems, etc., etc. have NFS implementations
of various sorts (some client-only, some server-only, some both - and if
you guessed "client-only" for DOS, you're wrong, there *do* exist server
implementations...).

The multiplicity of file system types is the reason why NFS makes
reading a directory an operation that returns "canonicalized" directory
entries rather than the raw directory data, and why SunOS modified the
"readdir()" routine to use a "read canonicalized directory entries"
calls rather than "read()".  (As indicated, making "read()" on a
directory return canonicalized entries might have been another way of
doing that.)

>>VMS's editors do, I think, support line numbers - but those aren't keys.
>
>Well, the difference in my mind is that line numbers of later lines
>change when you insert lines before them, whereas keys don't. How do
>VMS's line numbers work?

They may change, or they may not.  According to an EDT manual lying
around here, EDT will try to use decimal fractions for line numbers in
order to avoid renumbering lines, but "in extreme cases EDT may be
forced to renumber lines after the last line you insert".  I assume
those line numbers become the "fixed control" portion of the line if the
file is in "variable with fixed control" format.  I don't see anything
in the manual that indicates what happens if you insert stuff between
line 17.1 and line 17.2, say.

>Lets take this as a definition:
>line -- a piece of information in a file that is of fixed size once written.
>record -- a piece of information in a file that can be replaced by a 
>       different-sized piece later.
>line-number [of a record] -- the ordinal number of lines or records before 
>	this line or record.
>key -- an arbitrary identifier associated with a line or record which can
>	be used for random access. Sometimes (usually in text editors) these 
>	keys are numbers, giving rise to confusion between keys and 
>	line numbers.

In that case, as far as I know, in RSX and VMS, text files consist of
lines, not records, and files don't have line numbers as keys, by those
definitions.  (Text files aren't random-access files.  You *might* be
able to replace a line with a shorter line, but not with a longer line,
and probably not even with a shorter line.  Line numbers are attributes
hung off lines, but aren't keys for random access.)

I've never seen a system that uses records, by that definition, as
lines, nor any that use line numbers as keys.  I presume you have.

>Consider this:
>Other than the directory management stuff, did switching from the V7
>disk layout to the BSD4.x layout "somehow ... manage to *completely*
>hide the management of keys [in this case, file names] from the
>program"?  Yes, because every program *was* written to use the calls
>(open(), ...) that manage the "keys" (in this case, file names).
>Programs that *didn't* use the calls (fsck) had to be rewritten.

I.e., not *every* program was written to use the calls.  "fsck" wasn't,
but it's a special case; it manipulates raw file systems, rather than
files on a file system.  More relevantly, programs that read directories
weren't written to use calls that hide the implementation details of
directories from the program, because no such calls existed.  Nowadays,
programs are generally being written to use those calls, now that they
exist, so the programs need at most be recompiled when built on a new
system, and may not even need to be recompiled if you plug a new file
system type into a system.

>Imagine that SCCS was written like NFS instead of like tar.
>What would you have?

DSEE. :-)

Apollo's OS has the mechanism I described earlier, and one of the uses
they made of it was to implement SCCS-oid files as "file types", so that
the ordinary "stream I/O" read operations on an SCCS file would return
the text of the latest version of the file, by default (I think there's
a way of having it give some other version, perhaps by setting something
like an environment variable).

I'm not sure what'd happen if you did a "write this file out"
operations; it might create a new version.  Of course, that might have
the problem that it wouldn't know where to get the history comments for
the new version of the file, so perhaps that's not how it works.  (This
is another case, like the case of keyed files, where having an abstract
"stream read" operation works a bit better than an abstract "stream write"
operation; the "stream read" operation generally only has to *discard*
data to make a more complexly-structured file look like a stream of
lines or whatever, while a "stream write" operation would have to *add*
data, and few systems implement the Read User's Mind operation.)

>True. However, if you are writing a file from scratch and the standard
>text editor normally numbers a file from 100 in steps of 10, then maybe
>the interface COR could "magically" chose the same defaults.
>
>In reality, I've used systems where you could not open a keyed file
>for writing in consecutive mode, for exactly the reason you stated above.
>In practice, this wasn't a problem.

It wasn't a problem because text files weren't keyed files, or it wasn't
a problem because all the editors knew about keys?

>>>One of my complaints with the UNIX file system is that none of the tools
>>>that make UNIX useful can handle any of the complexly-structured files
>>>I need to build. If the mechanism for complex files was in place from
>>>the start, then all the tools would handle them.
>>I don't believe that - you'll have to prove it.  As indicated, that
>>requires that "all the tools" know about any stuff that can't be hidden
>>in the library interface.
>
>Look at the macintosh.  Every program I know of can handle the resource
>fork. Sometimes it isn't visible to the user, but if you open any font,
>you have just used a complexly-structured file. Even the code segments
>are in the resource fork, just like VM paging is in UNIX. Saying that
>tools on the Mac don't all use "the mechanism for complex files" is like
>saying that not all programs under UNIX use virtual memory.
>
>I think I've sufficiently "proven" my point by an example of an OS
>where the complex file systems were there from the start.

Well, maybe.  What happens if - let's use the example you chose of "the
tools that make UNIX useful", under the assumption that many of them are
programs whose output is a character-stream or byte-stream file - you
need to have one of those programs generate a file with stuff in the
resource fork?  Would it be possible to have some Mac C implementation's
implementations of "write()" or of the standard I/O library routines
generate the right stuff to shove in the resource fork, and would that
be the case for *all* of the tools?  Or would some tools have to be
modified to put "non-default" stuff in the resource fork, or to put
anything there at all?

Again, this sounds like a case where providing a "stream of lines"
read interface to a structured file is easy, but providing a "stream of
lines" write interface to a structured file isn't so easy.

(Also, are there any cases where the tool would really *want* to know
about the extra structure?)

>>(In other words, can you use PIP to copy an ISAM file, and have the
>>copied file usable as an ISAM file?)
>
>I would hope so.  Maybe I have not used any *really bad* implementations
>of a complex file system.

I don't know which systems let you do that - or how many of them
accomplish that by not letting you add your own *new* structured file
types to the system.  (Note that the latter tends to require that
*somewhere* in the system there's a way of getting at the "raw bits" of
the file.  I think the Apollo system used memory-mapping to do that,
sort of as if "read()" and "write()" and the like were implemented atop
"mmap()".)

>>How are arbitrary programs that *generate* text files to assign keys to
>>them, or is it the case that there's no *need* for them to generate
>>keyed files?
>
>I would think that if the human user is not assigning keys (like
>compiler error message listings) then unkeyed files would be OK.
>This assumes there would be a utility to put the "standard" keys for
>the text editor onto the file.

Yup, but now we're back to "tar" managing "tar" files, and SCCS having
special utilities to manage SCCS files, and....

>Umm... the part that isn't an application?  I'm not sure where
>user open-time CORs come into this. Basically, I don't knwo how
>to define this cleanly and will attempt to avoid "kernal" in the
>future.

Good idea; in SunOS 4.x/S5R4, and probably in other systems, stuff bound
to at run-time needn't live in the kernel.  It's conceivable that you
could re-implement "read()" and "write()" in userland, perhaps atop
"mmap()" (and perhaps with the object-oriented file system notions
mentioned before) - and even have existing dynamically-linked binaries
run without change.

>In the case where user-mode open-time CORs get attached to the
>application when the application opens the file, the CORs are not part
>of the privledged supervisor but are part of the filesystem. In this
>case, the kernel calls the usermode CORs,

Not necessarily.  In the example in my previous paragraph, the
application would directly call the usermode routines, which would then
ultimately call routines that trap into the kernel in order to actually
get a file descriptor for the file, or map it into the process's address
space, or whatever.

>which define in part the interface between the application and the
>stored data (the "filesystem"), but clearly the CORs are not part of
>the privledged supervisor. Here, parts of the filesystem lie outside
>the kernel but are presumedly (for security) accessible only through
>kernel calls.

I wouldn't necessarily presume that; in some systems, the "abstract"
file access routines (the ones providing the interface that most
programs see) are completely unprivileged; while this does permit a
program running with sufficient privileges to write a file to destroy the
structuring of the file, it doesn't permit a program *not* running with
those privileges from modifying the file in any way.  (That is, as far
as I know, the case with RSX; it may be the case with VMS, unless you
can't QIO a write on top of an RMS file outside of executive mode.)

If you want to have that level of security, I might prefer to have a
fairly general mechanism to provide it.  You may forbid arbitrary
unprivileged code that has write permission on a keyed file from
destroying the key structure, but that wouldn't necessarily forbid a
program from doing keyed accesses to, destroy some higher-level
structure above the key structure.  E.g., if the file is a database of
routines and variables in some large program, you could enter the name
of a non-existent routine into it and say it lives in some non-existent
source file or some source file that actually doesn't contain that
routine.

How much protection do you want to enforce with that level of security?
Do you provide a mechanism that basically encapsulates arbitrary
abstract data types and prevents any code other than the type manager
from accessing the representation of that data type (barring, of course,
compiler, hardware/microcode, privileged software, etc. bugs)?  Or do
you "draw the line" somewhere and say "stuff below the line is
protected, stuff above the line isn't"?  (I'm not advocating or
attacking either technique - there are costs and benefits to both - I'm
just asking.)