Re: Occassional "permission denied" in the middle of a large transfer over NFS

Vincent Hoffman wrote:
> On 01/07/2012 01:53, Rick Macklem wrote:
> > Vincent Hoffman wrote:
> >> Just a note to say I have tested this on -CURRENT with the new nfs
> >> server and it is still the case.
> >>
> >> On the client (FreeBSD seaurchin 8.3-RELEASE-p3 FreeBSD
> >> 8.3-RELEASE-p3
> >> #0: Tue Jun 12 00:39:29 UTC 2012
> >> root_at_amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
> >> amd64)
> >>
> >> [root_at_seaurchin /mnt/nfs/vm]# date ; tar -cf foo.tar
> >> /var/nfsen/profiles-data/ ; date
> >> Thu Jun 28 20:12:36 BST 2012
> >> tar: Removing leading '/' from member names
> >> tar: Write error
> >> Thu Jun 28 20:12:38 BST 2012
> >> [root_at_seaurchin /mnt/nfs/vm]#
> >>
> >>
> >>
> >> on the Server
> >> [root_at_fbsd /var/tmp]# uname -a ; date ; while true ; do mount
> >> /dev/md0
> >> /mnt/tmp/ ; umount /mnt/tmp ; done
> >> FreeBSD fbsd 10.0-CURRENT FreeBSD 10.0-CURRENT #22 r237646: Wed Jun
> >> 27
> >> 19:13:26 BST 2012
> >> toor_at_fbsd.bmk.namesco.net:/usr/obj/usr/src/sys/unsane-vm amd64
> >> Thu Jun 28 20:12:38 BST 2012
> >> ^C
> >> [root_at_fbsd /var/tmp]#
> >>
> >> any suggestions welcome.
> >>
> >> Vince
> >>
> >>
> >> On 27/06/2012 16:27, Vincent Hoffman wrote:
> >>> Hi,
> >>> After only one off-list reply from the author of kern/136865 (see
> >>> below)
> >>> after asking -questions, I thought it worth asking -CURRENT.
> >>> Basically:
> >>>
> >>> I seem to have run into the problems described in this old thread.
> >>> http://lists.freebsd.org/pipermail/freebsd-questions/2004-April/044927.html
> >>>
> >>> tl:dr mountd may give incorrect permission denied errors when it
> >>> is
> >>> refreshing the exports list due to non-atomic operations,
> >>> /sbin/mount has code that sends SIGHUP to
> >>> mountd on any mount operation, which implies that any manual mount
> >>> request would cause the problem.
> >>>
> >>> Currently I have still only tested on 8.3-RELEASE but the svn log
> >>> doesnt
> >>> seem to mention a fix since then. I'm currently taking a VM up to
> >>> -CURRENT to test.
> >>>
> >>> Looking though old PRs I see the following related.
> >>> kern/131342
> >>> kern/136865 (with patch for 7.2 and links to
> >>> http://nfse.sourceforge.net/ for -CURRENT )
> >>>
> >>> Does anyone who is qualified (sadly not me) feel like looking at
> >>> the
> >>> code to see if its suitable for inclusion in part/whole as not
> >>> having
> >>> NFS transfers interrupted by local mount operations on the nfs
> >>> server
> >>> would be very handy :)
> >>>
> > I haven't looked at Andrey's patch, but conceptually it sounds like
> > the best approach. As I understand it, the problem with replacing
> > mountd with nfse (at least in the FreeBSD source tree) is that nfse
> > is not 100% backwards compatible with /etc/exports and, as such, is
> > a POLA violation.
> Understood. Its far from a simple drop in replacement.
> >
> > To modify mountd to use the kernel changes is more work than I have
> > time for, in part because mountd.c is a very ugly old piece of C
> > code, imho.
> >
> > I do have a patch that suspends/resumes execution of the nfsd
> > threads
> > (new, experimental for FreeBSD8.n only) when mountd reloads
> > /etc/exports.
> > This approach has certain disadvantages vs Andrey's transactional
> > load/commit
> > model, but it is simple and might be sufficient for your problem.
> > If you want to try this patch, it is at:
> >    http://people.freebsd.org/~rmacklem/atomic-export.patch
> > (The patch affects both the kernel and mountd.c.)
> Happy to try it, to be certain I have understood, this is for the
> newnfs
> server (experimental in 8.x default for 8
> 9+)
Yes, that is correct. For the old (default on 8.x) it will have no effect.

> I'll apply to my -CUREENT test VM for now. Will fire up a 8.3 vm when
> i
> get a minute.

Also, to enable it, you must add a "-S" flag to mountd_flags, after you've
replaced the mountd executable.

The patch is pretty straightforward, but has not seen much testing.
Hopefully, it might be useful for you, rick

> >
> > Also, you could easily hack mount.c so that it doesn't send a SIGHUP
> > to mountd (which causes the exports to be reloaded) every time a
> > local
> > fs is mounted.
> True and I may have to do that for the production NAS for the time
> being.
> Thanks for looking at this.
> 
> Vince
> >
> > rick
> >
> >>> thanks, Vince
> >>>
> >>>
> >>> _______________________________________________
> >>> freebsd-current_at_freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >>> To unsubscribe, send any mail to
> >>> "freebsd-current-unsubscribe_at_freebsd.org"
> >>
> >> _______________________________________________
> >> freebsd-current_at_freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >> To unsubscribe, send any mail to
> >> "freebsd-current-unsubscribe_at_freebsd.org"
> > _______________________________________________
> > freebsd-current_at_freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to
> > "freebsd-current-unsubscribe_at_freebsd.org"
> 
> 
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscribe_at_freebsd.org"