Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!ucbvax!decwrl!sun-barr!cs.utexas.edu!uunet!mcvax!ukc!strath-cs!jim
From: jim@cs.strath.ac.uk (Jim Reid)
Newsgroups: comp.sys.hp
Subject: Re: Backup over net (Was Re: NFS Super users?)
Message-ID: <253@baird.cs.strath.ac.uk>
Date: 31 Jul 89 15:06:39 GMT
References: <2924@helios.ee.lbl.gov> <63300007@hpl-opus.HP.COM>
Sender: news@cs.strath.ac.uk
Reply-To: jim@cs.strath.ac.uk
Organization: Comp. Sci. Dept., Strathclyde Univ., Scotland.
Lines: 84

In article <63300007@hpl-opus.HP.COM> jewett@hpl-opus.HP.COM (Bob Jewett) writes:
>> >  "find <nominiated file systems> -depth -print | cpio -oacxBC > /dev/rmt/0h"
>> 
>> This is not a clever idea. Recursive traverses of a remote file system
>> with something like find or ls -R will stress the fileserver to a high
>> degree. (Try it and watch the load average on the client and server go
>> through the roof while the perfomance of both gets so bad that they're
>> both unusable.)
>
>I just did try it.  The machine doing the find/cpio was an 835, the
>filesystem being traversed was NFS-mounted from a 350.  The test (somewhat
>unscientific) was the startup time for "vi /etc/hosts" (1.2 megabytes) on the
>350, which is also the fileserver for 15 other systems.  It took only about
>twice as long to start (30 seconds) as on an unloaded 350 (~15 seconds).
>
>On what sort of configuration do you have a response-time problem?

Oh, on pretty much any NFS client/server combination. It doesn't matter
who makes the clients and servers, let alone the h/ware configurations
of particular boxes. High-bandwidth I/O controllers and fast processors
can make things easier - but such machines can still be brought to their
knees by doing a du of a large (100 MByte+) remote file system.

>> It is much better to do this sort of thing on the machine where the
>> files physically live. This saves network bandwidth and servicing
>> gazillions of remote file system protocol requests and acknowledgements
>> that are typically done at interrupt time on both the client and server.
>
>We have found that the 835 can do a "du" on /nfs/server350 about twice as
>fast as the 350 can do a "du" on its own disks.

This does not make sense.

You must be seeing caching effects here. The server has to do the work
of fetching the information of disk regardless of whether the request
was local or remote. That work is a fixed overhead, no matter if the
reason for the work was locally generated (by a system call) or from a
remote client's NFS requests. Once a kernel is given that work to do, it
generates disk requests. It then leaves the data in the buffer cache, the
inode table and sundry other places where it can be picked up without
going to disk for it again. This data can then get passed back to a
local process by means of a system call return. For an NFS request,
the kernel has to encode some of the response packet with XDR, compose
an NFS reply packet and queue it for transmission by the ethernet
device driver. This is much more work than simply returning from a system
call. All other things being equal, a local system call should therefore
be much quicker than an equivalent NFS request. (Bear in mind that the
client has also to compose & transmit the NFS request packet beforehand.)

The times you have seen suggest to me two things: [1] Your machines have
a monstrous system call overhead. This is unlikely. [2] Your server was
able to satisfy your client's NFS requests from locally-cached data
structures without having to go out to disk to get the information. This
is much more likely, so the difference can be explained by 'cache hits'
on the server - e.g. stat-ing inodes already in the server's inode table -
that save the server from making disk requests.

You lose this cacheing effect when the fileserver is busy and is turning
over inodes and disk blocks at a high rate because of other I/O. [Try
running du on a bit of the filesystem that has more inodes than can fit
in the kernel's inode table. Read a few big files too, so that the
buffer cache is unlikely to contain anything useful like bits of the
disk inode list.]

When the client's request can't be satisfied from a cache, the server goes
out to the disk and you take a performance hit. Now that's not the end
of the story. While the server goes out to disk, the NFS server process
that deals with that request blocks, waiting for the disk request to
complete. It can't deal with another NFS request while it waits. If that
is repeated for the other NFS nfsd processes, the server will be unable
to deal with incoming client requests, though perhaps only for a few
milliseconds. Those requests get lost and eventually are retransmitted by
the clients, causing further overheads and delays.

This is why it is generally better to do local I/O than use some network
file system. For day-to-day usage, the performance impact can be lived
with - most users tend not to do *that* much I/O. Recursive directory
traverses and stat calls (typical of find and du) are a different story
because of the enormous amounts of NFS traffic they cause. That NFS
traffic will get processed at interrupt time, so if the NFS servers are
busy for long periods of time, users will see response times drop
because the CPU spends most/all its time processing NFS requests.

		Jim