Path: utzoo!mnetor!uunet!yale!husc6!ukma!david From: david@ms.uky.edu (David Herron -- Resident E-mail Hack) Newsgroups: comp.dcom.lans Subject: Re: NFS vs RFS (actually, vs Sprite and Andrew) Message-ID: <8641@g.ms.uky.edu> Date: 22 Mar 88 05:35:08 GMT References: <10370@ut-sally.UUCP> <720@uel.uel.co.uk> <1695@uoregon.UUCP> <45660@sun.uucp> <3af44ad7.b8ab@apollo.uucp> Reply-To: david@ms.uky.edu (David Herron -- Resident E-mail Hack) Organization: U of Kentucky, Mathematical Sciences Lines: 120 Keywords: NFS, ANDREW, TOCS, consistency In article <3af44ad7.b8ab@apollo.uucp> rees@apollo.uucp (Jim Rees) writes: >I've never used a large NFS system, but I'm currently using a system >that includes about 2000 machines sharing a single file system. Any >file name can be used interchangeably by any machine. I have a hard time >imagining how this would work with NFS. I'm not sure if we count as "large". We've got about 20-30 workstations sharing disk space from 6 servers -- and growing. >For example, we have a few hundred people in R&D. Almost all of them >will at some time want to look at the kernel sources. If you use NFS, >how does this work? Does everyone mount the file system containing the >kernel sources all the time? Do you just mount it when you want to >look at the sources? What happens when the sources move to a different >machine? Is it possible for 2000 clients to each mount a couple dozen >servers? How do you keep track of where the servers are? You could do it either way depending on the local preferences. Around here, we attempt to make this grand hoax that we've got one file system, and all the nfs participants keep everything that's nfs'able mounted at all times. Part of our /etc/rc stuff is a script /etc/rc.remote which does things like: if [ `hostname` != some.local.machine ]; then /etc/mount all the stuff that some.local.machine makes available to people fi It USED to be EXACTLY like that, we now have a scheme where we keep a central copy of all the configuration files. There is one file in this central place which is used to make the individual copies which actually are used on the other machines. Each line of this central file is marked for which host(s) it applies to. When things move around we edit this file, rdist it to each machine, and then reboot everybody. (If you ask me, having to reboot everybody is a bit drastic -- but then I didn't configure that end of our system). The current effect is like above. I don't think it would work to have 2000 clients mounting the same thing with NFS. I doubt that the kernal tables are able to handle this reasonably, if at all. But maybe I'm wrong about that. What's the size of Sun's internal network? >If I write a trip report, and want to send it to a large group of people, >I put it in a file and send the file name out via email (or news). Then >the people who want to look at it just read that file. How would you do >this with NFS? E-mail the whole thing? Send out the file name and let >people mount your disk as needed? Again, same deal as above. Oh, and simply tell 'em the file name. >How does uucp work with NFS? Does every machine mount the uucp lib and >spool, or is there a single "uucp machine" that I rlogin in to when I >want to uucp something? There's a "uucp machine" that you have to rlogin to when you want to uucp something. The problems are: 1. devices don't export -- when you do the uucp/uux command, it might want to immediately call up a uucico to make a call. BUT the /dev guys don't exist, so you CAN'T make the call. 2. locking -- NFS didn't support across-the-net locking of files until recent versions. Therefore, if flock()'s were being used anywhere this wouldn't actually lock anybody out. 3. different physicial machines -- NFS runs in a mixed environment. That is, our nfs is shared between uVaxIIen running Ultrix, Vaxen running Mt Xinu 4.3+NFS, and Sun's. All have different binaries and can't run each others binaries. 4. different machines have different pid's -- UUCP's lock files contain the pid of the locker so that you can check to see if the lock is still valid. (i.e. kill(0, pid) and test the return code). >This is only partly meant as a criticism of NFS. I really am interested >in how you folks with large NFS installations get your work done. Yeah. I'm interested in how it works at really large places too. I'm also curious at how Apollo's system solves those problems. For instance, it'd be wasteful to have your kernel sources physically stored on multiple machines, not to mention the headaches of maintaining consistency in those multiple copies. That means that you've got to have your 200 guys in r&d accessing the same machine to get to those kernal sources. (Substitute for kernal sources just about anything you care to name -- but needs to be accessed by a lot of people at the same time -- news for instance). How would ANY similar system avoid being bogged down considerably when lots and lots of people are accessing the same stuff? (Be it news articles or kernal sources). Around here we've got two sorts of things which are commonly accessed over NFS. One is the user files -- all our workstations are basically diskless. (The uVax2000's have a 40 meg disk, but that's used soley for the important parts of the os and for tmp and swap). The user files are stored over on servers (at the moment, 2). The other is the news articles, troff stuff (font files), and /usr/local commands. Our experience has been that the two main servers have the most consistently high load averages as well as the most consistently high error rates in the ether drivers. (These servers are uVaxIIen with DEQNA's). The correlation between load average and nfs activity is easy to see, often if the load average is 3 then 3 of the nfsd's will be active. What do other systems do to avoid bogging down the source machine for heavily used items? The other issue with across-network file systems is coordinating the placing of objects within the global file system. At least with NFS and RFS the administrators have a lot of options on how they can arrange things, but then they end up with a couple of little files to edit on each machine whenever something changes. I've described our technique for getting around the hassle of making the same change to the same file on 20+ machines. What are other methods? -- <---- David Herron -- The E-Mail guy <---- or: {rutgers,uunet,cbosgd}!ukma!david, david@UKMA.BITNET <---- <---- "Oh, I dunno -- I think Sean would be rather tasty!" -- Becky