Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!uwm.edu!uakari.primate.wisc.edu!gem.mps.ohio-state.edu!apple!sun-barr!newstop!sun!commuter!beepy From: beepy%commuter@Sun.COM (Brian Pawlowski) Newsgroups: comp.protocols.nfs Subject: Re: NFS client has out-of-date files Summary: client caching, consistency semantics Message-ID: <125974@sun.Eng.Sun.COM> Date: 7 Oct 89 01:24:24 GMT References: <17787@bellcore.bellcore.com> <1967@convex.UUCP> <1989Oct6.203732.12847@acd4.UUCP> Sender: news@sun.Eng.Sun.COM Distribution: na Lines: 97 > >>A user was working on two Suns simultaneously. He was editing one file > >>on one Sun, and reading the file using LaTeX on the other. The file > >>was NFS mounted on both Suns. It physically resided on the NFS server, > >>a Pyramid. One of the Suns had an out-of-date copy! It was a minute > >>old. > > > >We've had this; what seems to be a common cause for it is that the > >time is not synchronized between the updating client and the server, > >so the file attributes don't get through the server and show up on > >disk until something changes. > > Well, here goes an attempt to describe what's happening. We had this > problem with our systems, and because of it have had to abandon using > NFS for our customer systems (at least for now). I'm curious what the application was that you had to abandon using NFS, and what the systems were. Your explanation was cogent, I'll babble on along the same lines. Not to be a pain, but it's not so much a "problem" as a implementation behaviour. The caching with consistency checks (typically on the order of every 30 seconds) introduces a window where a client's modifications to a file are not noticed by other client's for the cache consistency check (which consists of a NFS GETATTR call to inspect the modified time of the file to see if the cached file data is still valid). Unfortunately these consistency checks are not documented very well, and possibly differ from implementation to implementation. > The problem you are seeing is that the client can take too > long to realize the file has changed. There is a tradeoff here between performance and consistency. Actually, cache check intervals in our current NFS implementation differ for directories and plain files. Directory timeouts are: 30 - 60 seconds, File timeouts are: 3 - 60 seconds. In the NFSSRC 4.0 reference port acregmin=n Hold cached attributes for at least n seconds after file modification. acregmax=n Hold cached attributes for no more than n seconds after file modifica- tion. acdirmin=n Hold cached attributes for at least n seconds after directory update. acdirmax=n Hold cached attributes for no more than n seconds after directory update. actimeo=n Set min and max times for regular files and directories to n seconds. Regular defaults are: fg,retry=10000,timeo=7,retrans=3,port=NFS_PORT,hard,\ acregmin=3,acregmax=60,acdirmin=30,acdirmax=60 These values are bounded in the kernel between the above values. The reasoning in being able to set the file timeout lower than 30 seconds is to allow stronger (more frequent) consistency checks. In an internal version of NFS, there is also an option to turn off caching on a mount point. Caching is important to performance. NFS introduces a window of inconsistency defined by the timeout period. Other distributed filesystems (Sprite, Andrew) use call backs to ensure data consistency. Another option one might consider to ensure consistency of single writer multiple readers (or multiple writers multiple readers) is to use locking. If you have a custom app. you might try it. This would force exact consistency and proper serialization of readers and writers. > This is controlled by the sync/update > procedure, and can cause further delays of up to 30 seconds. (NFS > *might* be a write-through cache, but I don't think so.) So, with a 30 second sync update on the writer, and a 30 second default cache check window on the reader, you get: 60 seconds (bingo). > > Anyway, hope this helps anyone who has noticed the same problem. > Normally, it should not cause serious problems, especially if you keep > the system times synchronized. If you aren't expecting it though, it > can be quite frustrating. > Hmmmm... yeah I guess it can be a problem - especially if you're not expecting it. But it is a performance attack for NFS. Cache consistency checking on a more frequent basis would load both the network and the server with unnecessary checks for what I believe for most circumstances; that is synchronized writers and readers is an infrequent case in most applications. Think about it: I typically work on my own files, in my own area, and when I share files for program development, I use SCCS to synchronize access and manage the software. Many files accessed through NFS are read-only (shared executables). Locking is available in the case requiring synchronization and tight consistency. Brian Pawlowski Brian Pawlowski Sun Microsystems, NFS Development