Re: repeatable ZFS panic: share->excl

This is similar to the patch I've asked lulf_at_ to test except that it  
is longer and I fix a bug where zfs_lookup() can leak a vnode lock if  
the access check fails. :-)  The last one I sent to lulf_at_ is at www.FreeBSD.org/~jhb/patches/zfs_ea.patch 
.

-- 
John Baldwin

On Mar 12, 2009, at 11:35 PM, Attilio Rao <attilio_at_freebsd.org> wrote:

> 2009/3/12, Anonymous <swell.k_at_gmail.com>:
>> Tim Kientzle <kientzle_at_freebsd.org> writes:
>>
>>> Peter Schuller wrote:
>>>>>  First problem I notice is a panic, which seems to occur every
>>>>> time 'tar' is used. Might be unrelated to tar, but it definately
>>>>> provokes it. Simply the tar command or a pkg_add causes it.
>>>>
>>>> See the "ZFS/extattr lockup"/"bsdtar lockup on current" threads  
>>>> from
>>>> the past handful of days.
>>>
>>> I think this may be a different problem.
>>> The earlier thread involved a ZFS bug that causes
>>> it to lock up if it receives a request to enumerate
>>> extended attributes on a file (via extattr_list_link
>>> system call).  Tar recently added support for
>>> backing up extended attributes.  I've disabled
>>> that support until this particular ZFS problem can
>>> be fixed.
>>
>>
>> I guess you're wrong, it's same issue. Here is output from unmodified
>> kernel (r189728) under qemu which looks exactly as on that
>> screenshot. Perhaps, the panic is triggered by INVARIANTS.
>>
>> # lsextattr -h user foo
>> shared lock of (lockmgr) zfs _at_ /usr/src/sys/kern/vfs_lookup.c:477
>> while exclusively locked from /usr/src/sys/modules/zfs/../../cddl/ 
>> contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:152
>> panic: share->excl
>> cpuid = 0
>> KDB: enter: panic
>> [thread pid 105 tid 100078 ]
>> Stopped at      kdb_enter+0x3d: movq    $0,0x662538(%rip)
>>
>> db> bt
>> Tracing pid 105 tid 100078 td 0xffffff00015bea80
>> kdb_enter() at kdb_enter+0x3d
>> panic() at panic+0x17b
>> witness_checkorder() at witness_checkorder+0x16e
>> __lockmgr_args() at __lockmgr_args+0xd1b
>> vop_stdlock() at vop_stdlock+0x39
>> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
>> _vn_lock() at _vn_lock+0x57
>> lookup() at lookup+0xf4
>> namei() at namei+0x545
>> zfs_listextattr() at zfs_listextattr+0x18c
>> VOP_LISTEXTATTR_APV() at VOP_LISTEXTATTR_APV+0xb5
>> extattr_list_vp() at extattr_list_vp+0x22a
>> extattr_list_link() at extattr_list_link+0xc3
>> syscall() at syscall+0x1e7
>> Xfast_syscall() at Xfast_syscall+0xab
>> --- syscall (439, FreeBSD ELF64, extattr_list_link), rip =  
>> 0x800692e4c, rsp = 0x7fffffffed08, rbp = 0x7fffffffede0 ---
>>
>> db> show all locks
>> Process 105 (lsextattr) thread 0xffffff00015bea80 (100078)
>> exclusive lockmgr zfs (zfs) r = 0 (0xffffff0001581578) locked _at_ / 
>> usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ 
>> fs/zfs/zfs_znode.c:152
>> exclusive lockmgr zfs (zfs) r = 0 (0xffffff0001581a58) locked _at_ / 
>> usr/src/sys/kern/vfs_extattr.c:668
>>
>> db> show lockedvnods
>> Locked vnodes
>>
>> 0xffffff00015819c0: tag zfs, type VREG
>>    usecount 1, writecount 0, refcount 1 mountedhere 0
>>    flags ()
>>    v_object 0xffffff0001574898 ref 0 pages 0
>>    lock type zfs: EXCL by thread 0xffffff00015bea80 (pid 105)
>> #0 0xffffffff80530568 at __lockmgr_args+0x758
>> #1 0xffffffff805c0fd9 at vop_stdlock+0x39
>> #2 0xffffffff808557db at VOP_LOCK1_APV+0x9b
>> #3 0xffffffff805dd6a7 at _vn_lock+0x57
>> #4 0xffffffff805c2f20 at extattr_list_vp+0xb0
>> #5 0xffffffff805c3173 at extattr_list_link+0xc3
>> #6 0xffffffff8080f227 at syscall+0x1e7
>> #7 0xffffffff807ec39b at Xfast_syscall+0xab
>>
>> 0xffffff00015814e0: tag zfs, type VDIR
>>    usecount 1, writecount 0, refcount 1 mountedhere 0
>>    flags ()
>>    lock type zfs: EXCL by thread 0xffffff00015bea80 (pid 105)
>> #0 0xffffffff80530568 at __lockmgr_args+0x758
>> #1 0xffffffff805c0fd9 at vop_stdlock+0x39
>> #2 0xffffffff808557db at VOP_LOCK1_APV+0x9b
>> #3 0xffffffff805dd6a7 at _vn_lock+0x57
>> #4 0xffffffff8108405e at zfs_znode_cache_constructor+0x4e
>> #5 0xffffffff81085029 at zfs_znode_alloc+0x39
>> #6 0xffffffff81085915 at zfs_mknode+0x205
>> #7 0xffffffff810948f5 at zfs_make_xattrdir+0x155
>> #8 0xffffffff81095bc3 at zfs_get_xattrdir+0xd3
>> #9 0xffffffff810a4dbf at zfs_lookup+0x11f
>> #10 0xffffffff810a51d8 at zfs_listextattr+0x128
>> #11 0xffffffff80853255 at VOP_LISTEXTATTR_APV+0xb5
>> #12 0xffffffff805c309a at extattr_list_vp+0x22a
>> #13 0xffffffff805c3173 at extattr_list_link+0xc3
>> #14 0xffffffff8080f227 at syscall+0x1e7
>> #15 0xffffffff807ec39b at Xfast_syscall+0xab
>
> Could you please re-enable the extended attributes on bsdtar, try this
> patch and report to me?:
> http://www.freebsd.org/~attilio/zfs_vnops.diff
>
> (sorry if it is untested but I just don't have any ZFS machine).
> Actually, I think the problem is that zfs_lookup() doesn't get the
> correct handover once the extended attributes directory vnode is
> retrieved.
>
> Thanks,
> Attilio
>
>
> -- 
> Peace can only be achieved by understanding - A. Einstein