Path: utzoo!utgpu!jarvis.csri.toronto.edu!rutgers!cs.utexas.edu!uunet!mcsun!ukc!icdoc!qmc-cs!liam
From: liam@cs.qmc.ac.uk (William Roberts)
Newsgroups: comp.protocols.nfs
Subject: Re: NFS not idempotent (was re: mountd Performance under Stress)
Message-ID: <1211@sequent.cs.qmc.ac.uk>
Date: 6 Sep 89 16:59:34 GMT
References: <14068@bloom-beacon.MIT.EDU>
Reply-To: liam@cs.qmc.ac.uk (William Roberts)
Organization: Computer Science Dept, Queen Mary College, University of London, UK.
Lines: 55
Summary:
Expires:
Sender:
Followup-To:
Distribution:
Keywords:

In article <14068@bloom-beacon.MIT.EDU> jtkohl@athena.mit.edu (John T Kohl) writes:
>In article <1199@sequent.cs.qmc.ac.uk> liam@cs.qmc.ac.uk (William Roberts) writes:
>
>   This is a difference between user-level RCP and kernel-level RPC.
>   The kernel level *knows* that its NFS RPC requests are
>   idempotent
>
>Unfortunately, some of them ARE NOT idempotent, and that has caused
>great troubles to us at MIT.

The server implementations around keep a cache of recent
requests so that they can recite the previous reply if this
happens to be a retransmission. The previous reply will be
repeated IFF the cached xid matches the xid of the request,
which implies that the request is actually a retransmission.
This extra stuff is actually making the underlying RPC calls
idempotent, rather than the protocol requests themselves.

The requests for which this caching seems to be happening on my
systems (NFS 3.0) are:

    rfs_create, rfs_remove, rfs_link, rfs_mkdir, rfs_rmdir

>Consider:
>       create directory  (we've had mkdir return 'already exists' when
>                               it has actually just been created)
>       set attributes with length = 0 (truncate--with packet reordering
>                               and multiple biod's/nfsd's, this can lead to
>                               truncation occuring after a successive
>                               write()--poof, your file contents are gone)
>       rename
>       move

Move and rename are certainly a potential problem since their
answers aren't cached. I though file truncation was done using
rfs_create rather than rfs_setattr, but the multiple biods are
a problem since they can interleave the requests and indeed
give rise to a write being actioned before the truncation!
I can't see how mkdir can have done that to you (unless it was
that System V nonsense about "create ., create ..").

Question: What's the best way to fix the reordering problem?

My personal suggestion is to make "significant" operations such
as create act synchronously, so that the creating process
cannot issue a subsequent write request before the create has
definitely occurred.


-- 

William Roberts         ARPA: liam@cs.qmc.ac.uk
Queen Mary College      UUCP: liam@qmc-cs.UUCP    AppleLink: UK0087
190 Mile End Road       Tel:  01-975 5250
LONDON, E1 4NS, UK      Fax:  01-980 6533