Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!know!zaphod.mps.ohio-state.edu!brutus.cs.uiuc.edu!apple!sun-barr!ccut!titcca!cc.titech.ac.jp!necom830!mohta
From: mohta@necom830.cc.titech.ac.jp (Masataka Ohta)
Newsgroups: comp.arch
Subject: Re: fork and preallocation (was Re: Paging page tables)
Message-ID: <5894@titcce.cc.titech.ac.jp>
Date: 19 Jul 90 08:03:15 GMT
References: <920@dgis.dtic.dla.mil> <5830@titcce.cc.titech.ac.jp> <5DL4SPD@xds13.ferranti.com> <5855@titcce.cc.titech.ac.jp> <58184@bbn.BBN.COM> <5870@titcce.cc.titech.ac.jp> <58227@bbn.BBN.COM>
Sender: news@cc.titech.ac.jp
Organization: Tokyo Institute of Technology
Lines: 158

In article <58227@bbn.BBN.COM> lkaplan@BBN.COM (Larry Kaplan) writes:

>>The situation is well under control. Important processes can be programmed
>>to try mallocing several times with exponential back-off. Even if it dies,
>>it can cleanup its environment.

>This requires significant modification of programs.

No. All you have to do is redefine malloc and realloc.

>My proposals have not 
>required any change to applications, only additional system code.  A general
>solution does not require old applications to be rewritten.

But your proposal is not a solution.

>>Yes, it is always possible to resolve deadlock by human intervention.

>Don't put words in my mouth.  I say below that you can write a daemon
>(which is a program) to do this.  If you want to let a human do it, you can,
>but that is suboptimal.

You may claim you can write a perfect AI, which I don't mind.

>>It is very strange that you have reserved filesystem available for
>>swapping. You should have already allocated such a free space in advance
>>as swap area.

>This is only a convenience.  Depending where you implemented this code, you 
>could simply check a high water paging mark as mentioned by others.  Holding 
>some portion of your swap space in reserve is just another way to give you the 
>opportunity to play these games when trouble starts.  Using a high water mark
>can work just as well.

But I don't want to play with trouble. I would rather play nethack, instead.

>Read the posting.  It says daemon.  This means a program.

Oops! A program? Are you joking? What if the program itself run short
of pages?

>It could also be part of the kernel.

More reasonable proposal. It can be a dirty and unreliable workaround.

>>But it dose not worth doing so. Use vfork.

>This is your opinion (if I understand the sentence).  Making statements like 
>this does not address the other benefits that many people seem to like 
>about COW fork.

I will use COW fork, of course, if it exists and it is necessary. But,
to fork-exec something, you should use vfork.

>>>People may complain that this is not a truly
>>>general solution, and I would agree.

>>Vfork is the true solution.

>True?? By whose standards.  This is what we are debating.  Making statements
>like this without justification is worthless.

I showed several justification why simple fork is not appropriate
for fork-exec.

On the other hand, you showed no justification not to use vfork.

You only showed it is inelegant not to use vfork and insists on using
fork.

>There are most certainly situations where vfork() is not appropriate.

Have you shown any justification to claim so? Or, is it merely your
desire?

>>>As a side note, on the large systems I work on, we don't do preallocation
>>>and have never run out of paging (swap) space.  This is not to say that
>>>we never will, but typical systems have on the order of at least 10 times as
>>>much disk storage as main memory.  In some cases, as much as 100 times more.

>>You have 100 times more swap space because you think it may be filled,
>>don't you.

>NO.  These numbers come from simply looking at the systems that people have 
>running.  These numbers are true for most all computers in general.

You want to say most all computers in general have 10 to 100 times more
swap space than real memory? That is simply incorrect.

Very few system have 10 times more swap space.

>It may be 
>that most people can't page on all their filesystems, but thats not what I 
>said.  

Then, what you want to say?

You should make your point clear.

>>>I claim it is hard to fill that much disk
>>>space with paging and swapping traffic and still have a usable system.  You'll
>>>probably be thrashing to death long before that.

>>As you may know, programs manipulating large arrays, if written properly,
>>can use very large virtual space with little real memory without
>>thrashing. That is why some of your system are configured 100 times
>>more swap space, isn't it?

>You are attempting to justify the reasons for the way I have my systems 
>configured, when in reality this is the way just about everybody's systems are 
>configured.  

The thread of the vfork discussion begins because I said your system without
vfork is broken.

Moreover, it is you, who brought the configuration of your system into
the discussion.

So, why can't I refer to your system?

>The point about sparse matrix programs may be true to some extent.

Sparse matrix? I am not reffering to such a thing.

>These programs may or may not represent a significant portion of a machine's
>workload and therefore have some bearing on the techniques selected.
>Depending on the reference patterns of the programs though, if you really don't
>reference parts of the matrix at all, then pre-allocation of swap space is
>going to prevent the program from running when it would run fine without 
>preallocation.

If you don't have to reference parts of a matrix, there is an effecient
algorithm to do it without extra swap space.

>Even just reads from parts of the matrix wouldn't require
>swap space allocation since the pages aren't dirtied.  This sounds quite
>likely.

You can do so with mmap specifying readonly option. A cleaver implementation
won't allocate extra swap space.

BUT, such a thing has nothing to do with fork nor vfork.

>On inspection, it appears that one of our big systems currently in the field
>has 1 gigabyte of physical memory and only about 6.4 gigabytes of disk storage.
>They run sparse matrix problems.  They run lots of scientific codes.  They have
>yet to have any problems with swap space.  

You might have been lucky, or, you might have just overlooked the
problem.

>It may be that the 100 number is a little exaggerated.  Having disk storage 
>being 10 times the amount of physical memory in large machines seems to be 
>more the rule.  Disk servers, however, move more towards the 100 mark.

So what?

						Masataka Ohta