Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!mailrus!purdue!gatech!hubcap!jw
From: jw@sics.se (Johan Widen)
Newsgroups: comp.parallel
Subject: Re: distributed make(1)?
Message-ID: <5492@hubcap.clemson.edu>
Date: 16 May 89 12:57:16 GMT
Sender: fpst@hubcap.clemson.edu
Lines: 51
Approved: parallel@hubcap.clemson.edu

>Can anyone give me a pointer to literature on, or even an ftp
>target for, a distributed version of make?  GNU make spawns

Adam de Boor wrote a program called pmake (for parallel make), while he was at
Berkeley. It will distribute jobs between a set of machines "of the same
architecture".

Pmake actually relies on a set of separate programs to do the load balancing.
These programs can be used by any software, not just pmake.

I found a version of this program on a tape and has since fixed a number
of bugs (most are related to pmake being developed under Sprite, while we
run SunOS 4). We have it installed on Sun-3's and Sun-4's under SunOS 4,
including the loadbalancing.

We also have it running on a Symmetry, with local parallelism (a la GNU
make) only. Pmake uses up a couple of file descriptors for each
child it forks off locally. As the Symmetry OS (DYNIX) allows only 20 open
files per process (sigh), this puts a fairly severe limit on the parallelism.

The loadbalancing scheme has some interesting features. Each machine is
assigned to a class. Work is only shared between machines belonging to
the same class. 'Import of work' can be further restricted by explicitly
specifying which machines may export work to our machine. A machine may
decide if it is willing to import work depending on 0 or more of: how long
the machine has been idle (since someone touched the keyboard), the
load average, the amount of free swap space.

A nit in our current version of pmake is that the output from the children
to the screen seems to be block buffered rather than line buffered.


I have found that pmake gives a substantial speedup, especially if
the machines have a reasonable amount of memory (ie more than 4 MBytes).

Still, I tend not to use it very often. The reason is that many Makefiles
have been written with no thought given to parallelism, they often rely
on quirks in the standard make. Another problem is the NFS file caching:
it takes time for a machine to discover that another machine has modified
a file. These problems are usually easily dealt with when you write new
Makefiles, but they can be a real pain when you are confronted with
an already existing source tree.


A version of pmake (archive/pmake.fix2.Z) can be retrieved from
	sics.se (192.16.123.90)

--
Johan Widen
SICS, PO Box 1263, S-164 28 KISTA, SWEDEN	Internet: jw@sics.se
Tel: +46 8 752 15 32	Ttx: 812 61 54 SICS S	Fax: +46 8 751 72 30