Path: utzoo!utgpu!news-server.csri.toronto.edu!bonnie.concordia.ca!uunet!auspex!guy
From: guy@auspex.auspex.com (Guy Harris)
Newsgroups: comp.os.misc
Subject: Re: What constitutes a good OS?
Message-ID: <5362@auspex.auspex.com>
Date: 18 Jan 91 19:20:29 GMT
References: <41907@nigel.ee.udel.edu> <5331@auspex.auspex.com> <42010@nigel.ee.udel.edu>
Organization: Auspex Systems, Santa Clara
Lines: 223

>I meant that you used the standard open() read() and close() routines,
>as opposed to readdir(), opendir(), etc.  Hence, you could use
>cat, sed, etc to look at and process directories,

Only to a limited extent.  See Sean Fagan's posting.  Directories
contained *B*I*N*A*R*Y* data back in V6/V7 days, and did *N*O*T*
necessarily have nice newlines in them, and there was *N*O* guarantee
that "sed" could understand them.  The fact that you could use "cat" was
irrelevant, unless you were using one of the "cat" flags that expected
the file to be a text file; I can use "cat" on "/vmunix" if I want to.

>which you can no longer do.

I can still use "cat" on directories on the local machine, although I
can't use the flags that expect the file to be a text file, nor use it
to display the directories in some nice easily-read format - but then I
couldn't exactly do that under V6/V7 either.

>But if the filesystem supports keys, then your editors probably do also.

I wouldn't assume that - VMS includes keyed file support, but I don't
think its editors support keys.

(I assume by "editor" you mean "text editor"; however, since by "text
file" you appear to have meant, in the discussion on whether directories
were plain text files or not, "files you can read with "open()",
"read()", and "close()", which is a rather non-standard definition, I'd
like some verification that you actually mean the same thing by "editor"
that many of the rest of us do.  If not, bear in mind that by using
words in the same way that others do, you make communication easier....)

VMS's editors do, I think, support line numbers - but those aren't keys.

>>So, in RSX-11, are keys in "the filesystem" or in "a library"?
>
>Both.  Why should I have to choose?

Because, in an earlier posting, you said:

	If keys are in the filesystem as opposed to a library,
	then every program has access to that information automatically.

which indicates that you do *NOT* consider them equivalent, and
therefore that keys *cannot* both be in "the filesystems" and in "a
library".

Please be more careful in the way you phrase things, and more consistent
between postings; it's hard to consider ideas - and I think a lot of
what you have to say is worth considering - when you have to spend
cycles interpreting the way they're phrased due to unusual use of
descriptive phrases, or have to shift mindset from posting to posting
because in posting A something in "a library" wasn't the same as
something in "the filesystem" while in posting B they were the same. 

>The real choice is between a user-mode library and the kernel, and
>maybe even that can be resolved.

OK, so now if keys are in "a library", a program may still have access
to that information automatically (*every program* doesn't, unless
somehow the library can manage to *completely* hide the management of
the keys from the program, or every program is written to use the calls
that manage keys), and having keys in "the filesystem" is no longer
opposed to having them in "a library"?

Choose one or the other; I refuse to accept a choice of both....

>*** The other choice (which is what you are actually asking) is a
>choice between a link-time library and an open-time library.  If I need
>to code special calls into my application to handle different formats
>of files,

What exactly do you mean by an "open-time library", and how does the
above have anything to do with the choice between a "link-time library"
and an "open-time library"?

>then I can't use new formats with old programs;

Exactly my point in the parenthetical note above - if you have to
explicitly make calls to manage keys, any application that you'd want to
manage keys would have to be written knowing about those calls.  Maybe
when *reading* a file, you can have an interface to get only the data,
not the keys, from a file; that doesn't work when writing out a new
file, unless your library can magically figure out what key should be
assigned to each unit of the file.

>for example, I can't use .PAG and .DIR files with old versions of 'cp'
>without expanding the actual size the files take by filling in the holes.

One way to solve that *without* teaching "cp" about ".dbm" files is to
have it check for all-zero blocks and write them out as holes.

>One of my complaints with the UNIX file system is that none of the tools
>that make UNIX useful can handle any of the complexly-structured files
>I need to build. If the mechanism for complex files was in place from
>the start, then all the tools would handle them.

I don't believe that - you'll have to prove it.  As indicated, that
requires that "all the tools" know about any stuff that can't be hidden
in the library interface.

>Even a simple mechanism whereby a portion of the middle of a file could
>be replaced by a different size portion (records), and preferable able
>to be named and retrieved in order (keys), would make many of my programs
>much easier.

Do you need that ability for *all* files, or would an ISAM file type -
such as you'd have were all UNIXes to come standard with ISAM libraries
- be sufficient?

>Actually, the RSX-11 method seems pretty elegant to me, from what I've
>heard friends say of it. (I've never actually programmed on it.)  Don't
>you think that the RSX-11 mechanism is more in the UNIX philosophy than
>the current UNIX filesystem?  More modular? More getting of stuff out
>of the kernel? More reusable tool-ish?

No, not necessarily.

They *did* happen to move the *low-level* file system (the thing that
manages a namespace containing "bags of bytes") out of the kernel, into
a separate process - but I'm told that VMS moved it back in (XQPs as
opposed to ACPs).

That is, however, not at all connected to the issue of the user-mode
file access libraries in RSX including support for keyed files as
standard vs. the user-mode file access libraries in UNIX (unless you
count "dbm") not including them, nor the issue of the RSX libraries
*perhaps* permitting the *same* calls being used to read, and perhaps
write, both formats.

(In other words, can you use PIP to copy an ISAM file, and have the
copied file usable as an ISAM file?)

>>Were all UNIX systems to come with an ISAM library - with the C-ISAM
>>programmatic interface, say - and were all UNIX systems to have a file
>>system that permits storing and retrieving of attributes the way
>>Files-11 does, and were all the ISAM libraries to use that, would keys
>>in "the filesystem" or in "a library"?
>
>A library. I would be possible and easier to write tools that did not
>use the library

It's *possible* to do that under RMS; just do your own QIOs.  It may be
a *pain* to do so, but I'm not sure that makes a difference.

>and hence were incompatible with any files written with
>the library. Therefore, most tools would be incompatible, just as many
>tools under UNIX are incompatible with binary files. If text-processing
>software (editors, compilers, etc) all made use of the keys, then I
>would be happy to use this mechanism.

*ONCE AGAIN*, and please *ANSWER* this question:

How are arbitrary programs that *generate* text files to assign keys to
them, or is it the case that there's no *need* for them to generate
keyed files?

The "standard" sorts of systems that provide that functionality do not
appear to have any way of letting it happen automagically.  The
"object-oriented" systems of the Domain/OS sort might do it, especially
if it's relatively easy to make a new class that inherits the methods
from "boring sequential text file" by inheriting methods from "keyed
file" and building the "boring sequential text file" operations from
them.

>It's kind of like having the paragraph string in VI, whereby you can tell
>vi what troff macros indicate a paragraph break. Sadly, vi can't tell
>what TeX macros indicate a paragraph break and which LaTeX macros
>indicate a paragraph break. If `paragraphs' were part of the
>filesystem, then vi, LaTeX, TeX, and everyone else would probably use
>this information and interoperate.

That depends on whether the filesystem's notion of what a "paragraph" is
matches what all those programs want.  Bear in mind that TeX has to run
on operating systems other than WonderfulOS (and that LaTeX is just a
TeX macro package and as such probably inherits *all* its notion of what
a "paragraph" is from TeX).

>>Again, the question of "what is 'the filesystem'?" comes to mind.
>
>In my mind, the filesystem is the interface between applications and
>the kernel for purposes of doing input and output operations.

OK, what does "the kernel" mean here?

Is it "the privileged supervisor"?

If so, *please* make up your mind - earlier *in this very posting to
which I'm responding* you indicated that code in a user-mode library
could be part of the file system.  Now you're saying it has to be in "the
kernel".  You can't have it both ways....

Is it "that part of the systems software that applications generally go
through to get at e.g. files"?  If so, that's a fairly different usage
of the term "kernel" than I've ever heard - and, frankly, I'd be tempted
to just call that part of the systems software "the filesystem", and
answer the question "what is 'the filesystem'?" that way....

Or is the problem that you really mean "in my mind, *in typical UNIX
implementations* the filesystem is the interface between applications
and the kernel for doing input and output operations"?  It sounds then
as if you're essentially complaining that UNIX has multiple levels at
which you can access files - you can use "read()" or "fgets()", for
example, to read a text file - and that not everybody uses the same
level.  Many applications use standard I/O for doing I/O, others go
straight to the "kernel" calls (which may or may not actually be in "the
kernel").

>>Now, in this system, what is "the filesystem"?  Is it the code - which
>>in this case, let us say is still in the kernel - that supports a
>>namespace, and "bags of bytes", with associated attributes, to which the
>>names in that namespace refer?  Or is it that code *plus* the
>>"open()"/"read()"/"write()"/etc. implementation, which supports text
>>files, keyed files, etc., etc., etc.?
>
>The latter.  Aplications doing an open() get the "correct" routine
>without contortions and without being able to bypass it without
>special work, just as BSD UNIX puts symbolic links in the filesystem
>which applications must do special work to bypass.  

OK, fine, so as of this paragraph, the filesystem is *NOT* the interface
between the application and the kernel, it's the interface that typical
applications use to get at files *regardless* of whether it runs in
privileged mode or not.