Re: repeatable ZFS panic: share->excl

Tim Kientzle <kientzle_at_freebsd.org> writes:

> Peter Schuller wrote:
>>>   First problem I notice is a panic, which seems to occur every
>>> time 'tar' is used. Might be unrelated to tar, but it definately
>>> provokes it. Simply the tar command or a pkg_add causes it.
>>
>> See the "ZFS/extattr lockup"/"bsdtar lockup on current" threads from
>> the past handful of days.
>
> I think this may be a different problem.
> The earlier thread involved a ZFS bug that causes
> it to lock up if it receives a request to enumerate
> extended attributes on a file (via extattr_list_link
> system call).  Tar recently added support for
> backing up extended attributes.  I've disabled
> that support until this particular ZFS problem can
> be fixed.

I guess you're wrong, it's same issue. Here is output from unmodified
kernel (r189728) under qemu which looks exactly as on that
screenshot. Perhaps, the panic is triggered by INVARIANTS.

# lsextattr -h user foo
shared lock of (lockmgr) zfs _at_ /usr/src/sys/kern/vfs_lookup.c:477
while exclusively locked from /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:152
panic: share->excl
cpuid = 0
KDB: enter: panic
[thread pid 105 tid 100078 ]
Stopped at      kdb_enter+0x3d: movq    $0,0x662538(%rip)

db> bt
Tracing pid 105 tid 100078 td 0xffffff00015bea80
kdb_enter() at kdb_enter+0x3d
panic() at panic+0x17b
witness_checkorder() at witness_checkorder+0x16e
__lockmgr_args() at __lockmgr_args+0xd1b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0xf4
namei() at namei+0x545
zfs_listextattr() at zfs_listextattr+0x18c
VOP_LISTEXTATTR_APV() at VOP_LISTEXTATTR_APV+0xb5
extattr_list_vp() at extattr_list_vp+0x22a
extattr_list_link() at extattr_list_link+0xc3
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (439, FreeBSD ELF64, extattr_list_link), rip = 0x800692e4c, rsp = 0x7fffffffed08, rbp = 0x7fffffffede0 ---

db> show all locks
Process 105 (lsextattr) thread 0xffffff00015bea80 (100078)
exclusive lockmgr zfs (zfs) r = 0 (0xffffff0001581578) locked _at_ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:152
exclusive lockmgr zfs (zfs) r = 0 (0xffffff0001581a58) locked _at_ /usr/src/sys/kern/vfs_extattr.c:668

db> show lockedvnods
Locked vnodes

0xffffff00015819c0: tag zfs, type VREG
    usecount 1, writecount 0, refcount 1 mountedhere 0
    flags ()
    v_object 0xffffff0001574898 ref 0 pages 0
    lock type zfs: EXCL by thread 0xffffff00015bea80 (pid 105)
#0 0xffffffff80530568 at __lockmgr_args+0x758
#1 0xffffffff805c0fd9 at vop_stdlock+0x39
#2 0xffffffff808557db at VOP_LOCK1_APV+0x9b
#3 0xffffffff805dd6a7 at _vn_lock+0x57
#4 0xffffffff805c2f20 at extattr_list_vp+0xb0
#5 0xffffffff805c3173 at extattr_list_link+0xc3
#6 0xffffffff8080f227 at syscall+0x1e7
#7 0xffffffff807ec39b at Xfast_syscall+0xab

0xffffff00015814e0: tag zfs, type VDIR
    usecount 1, writecount 0, refcount 1 mountedhere 0
    flags ()
    lock type zfs: EXCL by thread 0xffffff00015bea80 (pid 105)
#0 0xffffffff80530568 at __lockmgr_args+0x758
#1 0xffffffff805c0fd9 at vop_stdlock+0x39
#2 0xffffffff808557db at VOP_LOCK1_APV+0x9b
#3 0xffffffff805dd6a7 at _vn_lock+0x57
#4 0xffffffff8108405e at zfs_znode_cache_constructor+0x4e
#5 0xffffffff81085029 at zfs_znode_alloc+0x39
#6 0xffffffff81085915 at zfs_mknode+0x205
#7 0xffffffff810948f5 at zfs_make_xattrdir+0x155
#8 0xffffffff81095bc3 at zfs_get_xattrdir+0xd3
#9 0xffffffff810a4dbf at zfs_lookup+0x11f
#10 0xffffffff810a51d8 at zfs_listextattr+0x128
#11 0xffffffff80853255 at VOP_LISTEXTATTR_APV+0xb5
#12 0xffffffff805c309a at extattr_list_vp+0x22a
#13 0xffffffff805c3173 at extattr_list_link+0xc3
#14 0xffffffff8080f227 at syscall+0x1e7
#15 0xffffffff807ec39b at Xfast_syscall+0xab

>
> However, pkg_add uses tar to extract archives,
> not create them, so this is not the same
> problem.  I haven't disabled tar support for
> restoring extended attributes, but no
> existing packages should have extended attributes
> to be restored, so I doubt this is the same issue.
>
> 1) Do you get the same symptoms on UFS?
>
> 2) Is this on extraction or just create?
>
> 3) Can you run tar under "ktrace" and
> figure out what system call is provoking
> the problem?
>
> Tim