Re: LOR between NFS and syncer

On Sat, Jun 06, 2009 at 03:32:01AM +0200, Dag-Erling Sm??rgrav wrote:
> Anyone seen this one before?
> 
> NFS+ZFS server, diskless NFS client.  Both run head.  On the client:
> 
> lock order reversal:
>  1st 0xc301c164 syncer (syncer) _at_ /usr/src/sys/kern/vfs_subr.c:1693
>  2nd 0xc322c058 nfs (nfs) _at_ /usr/src/sys/kern/vfs_subr.c:2083
> KDB: stack backtrace:
> db_trace_self_wrapper(c07477c6,c2bbfa18,c05921c5,c058310b,c074a615,...) at db_trace_self_wrapper+0x26
> kdb_backtrace(c058310b,c074a615,c2cb0828,c2cb0688,c2bbfa74,...) at kdb_backtrace+0x29
> _witness_debugger(c074a615,c322c058,c0759e62,c2cb0688,c0750f5e,...) at _witness_debugger+0x25
> witness_checkorder(c322c058,9,c0750f5e,823,0,...) at witness_checkorder+0x839
> __lockmgr_args(c322c058,80500,c322c074,0,0,...) at __lockmgr_args+0x797
> vop_stdlock(c2bbfb88,c2bbfb70,c0591f6b,c0750f5e,c073ced7,...) at vop_stdlock+0x62
> VOP_LOCK1_APV(c079d720,c2bbfb88,c0750f5e,c07b5100,c322c000,...) at VOP_LOCK1_APV+0xf3
> _vn_lock(c322c000,80500,c0750f5e,823,df,...) at _vn_lock+0x5e
> vget(c322c000,80500,c2f1f480,c6e,c2f35c94,...) at vget+0xb9
> vfs_msync(c2f35c94,2,c0750f5e,d67,c2f35c94,...) at vfs_msync+0xe7
> sync_fsync(c2bbfc7c,c301c10c,80400,c0750f5e,69d,...) at sync_fsync+0x17b
> VOP_FSYNC_APV(c07975c0,c2bbfc7c,c0750f5e,69d,c2f1f480,...) at VOP_FSYNC_APV+0xda
> sync_vnode(c093d8f0,c093d8dc,3e8,6cc,4e20,...) at sync_vnode+0x168
> sched_sync(0,c2bbfd38,c073fcec,334,c2ceea90,...) at sched_sync+0x273
> fork_exit(c05d8570,0,c2bbfd38) at fork_exit+0xb8
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0, eip = 0, esp = 0xc2bbfd70, ebp = 0 ---

We lock the covered vnode when doing mount or unmount for the non-root
filesystem. Since witness works by recording order by lock name, we get
the order of mounted on -> mounted from vnode type. Thus, any interleave
in the mount type would give us a LOR.

The vnode list for the mount point also includes syncer vnode, that
has a special fsync vop. Usual order is syncer vnode lock -> vnode
lock for the real vnodes. But, when filesystem is unmounted, we get
the order covered vnode -> any filesystem vnode during sync before
unmount. This one gives the LOR you reported, assuming you unmounted
nfs share mounted on the top of nfs share.

I think both issues cannot result in a deadlock.