-current panic (ZFS-related?): Sleeping thread ... owns a non-sleepable lock

Hi,

After updating sources to yesterday -current and upgrading ZFS pool
and filesystems I'm getting reproducible kernel panic during a build
on ZFS.

It looks like zfs_freebsd_getattr went to sleep while process was
holding kqueue lock (acquired in knote()). If someone needs additional
info, just email me ddb commands and I'd be happy to help.

Sleeping thread (tid 100247, pid 5591) owns a non-sleepable lock
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sx_xlock_hard() at _sx_xlock_hard+0x1a2
_sx_xlock() at _sx_xlock+0xa3
rrw_enter() at rrw_enter+0x98
zfs_freebsd_getattr() at zfs_freebsd_getattr+0xa5
filt_vfsread() at filt_vfsread+0x51
knote() at knote+0xfa
VOP_WRITE_APV() at VOP_WRITE_APV+0x153
vn_write() at vn_write+0x236
dofilewrite() at dofilewrite+0x7a
kern_writev() at kern_writev+0x52
write() at write+0x4d
ia32_syscall() at ia32_syscall+0x17c
Xint0x80_syscall() at Xint0x80_syscall+0x60
panic: sleeping thread
cpuid = 1
KDB: enter: panic
[thread pid 8466 tid 100199 ]
Stopped at      kdb_enter+0x3d: movq    $0,0x47dfd4(%rip)

db> trace 5591
Tracing pid 5591 tid 100247 td 0xffffff0039491720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21b
sleepq_switch() at sleepq_switch+0xfa
sleepq_wait() at sleepq_wait+0x46
_sx_xlock_hard() at _sx_xlock_hard+0x1a2
_sx_xlock() at _sx_xlock+0xa3
rrw_enter() at rrw_enter+0x98
zfs_freebsd_getattr() at zfs_freebsd_getattr+0xa5
filt_vfsread() at filt_vfsread+0x51
knote() at knote+0xfa
VOP_WRITE_APV() at VOP_WRITE_APV+0x153
vn_write() at vn_write+0x236
dofilewrite() at dofilewrite+0x7a
kern_writev() at kern_writev+0x52
write() at write+0x4d
ia32_syscall() at ia32_syscall+0x17c
Xint0x80_syscall() at Xint0x80_syscall+0x60

db> trace 8466
Tracing pid 8466 tid 100199 td 0xffffff000c0b9ab0
kdb_enter() at kdb_enter+0x3d
panic() at panic+0x176
propagate_priority() at propagate_priority+0x221
turnstile_wait() at turnstile_wait+0x212
_mtx_lock_sleep() at _mtx_lock_sleep+0xe0
_mtx_lock_flags() at _mtx_lock_flags+0xc3
kqueue_acquire() at kqueue_acquire+0x31
kern_kevent() at kern_kevent+0x6a
freebsd32_kevent() at freebsd32_kevent+0x69
ia32_syscall() at ia32_syscall+0x17c
Xint0x80_syscall() at Xint0x80_syscall+0x60

db> show alllocks
Process 31111 (bmake) thread 0xffffff013cd96000 (100385)
shared lockmgr zfs (zfs) r = 0 (0xffffff001e069098) locked _at_
/usr/src-8-mq/src/sys/kern/vfs_lookup.c:442
Process 31108 (as) thread 0xffffff000c261ab0 (100165)
exclusive sx user map (user map) r = 0 (0xffffff013cdb6360) locked _at_
/usr/src-8-mq/src/sys/vm/vm_map.c:3115
Process 31100 (cc1) thread 0xffffff003948f000 (100225)
exclusive sx user map (user map) r = 0 (0xffffff013ca5dc30) locked _at_
/usr/src-8-mq/src/sys/vm/vm_map.c:3115
Process 31063 (cpp) thread 0xffffff000c062390 (100106)
shared lockmgr nfs (nfs) r = 0 (0xffffff015a3527f8) locked _at_
/usr/src-8-mq/src/sys/kern/vfs_subr.c:2053
shared lockmgr nfs (nfs) r = 0 (0xffffff000cea3448) locked _at_
/usr/src-8-mq/src/sys/kern/vfs_subr.c:2053
Process 31017 (cc1) thread 0xffffff0043d31720 (100299)
exclusive sx user map (user map) r = 0 (0xffffff0039beb070) locked _at_
/usr/src-8-mq/src/sys/vm/vm_map.c:3115
Process 1138 (bmake) thread 0xffffff000c082390 (100162)
shared lockmgr zfs (zfs) r = 0 (0xffffff015a8c7270) locked _at_
/usr/src-8-mq/src/sys/kern/vfs_vnops.c:676
Process 55197 (sshd) thread 0xffffff0113652390 (100352)
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff00993825e8) locked _at_
/usr/src-8-mq/src/sys/kern/uipc_sockbuf.c:148
Process 1066 (bmake) thread 0xffffff0039d40000 (100285)
exclusive sleep mutex pmap (pmap) r = 0 (0xffffffff80993d80) locked _at_
/usr/src-8-mq/src/sys/amd64/amd64/pmap.c:2471
exclusive sleep mutex vm page queue mutex (vm page queue mutex) r = 0
(0xffffffff8097c580) locked _at_
/usr/src-8-mq/src/sys/amd64/amd64/pmap.c:2470
exclusive sx user map (user map) r = 0 (0xffffff00042a5070) locked _at_
/usr/src-8-mq/src/sys/vm/vm_map.c:2425
Process 11678 (sshd) thread 0xffffff000c087ab0 (100146)
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff001e02a5e8) locked _at_
/usr/src-8-mq/src/sys/kern/uipc_sockbuf.c:148
Process 5591 (bmake) thread 0xffffff0039491720 (100247)
exclusive sleep mutex kqueue (kqueue) r = 0 (0xffffff0039154500)
locked _at_ /usr/src-8-mq/src/sys/kern/kern_event.c:1635
exclusive lockmgr zfs (zfs) r = 0 (0xffffff001ec5f7f8) locked _at_
/usr/src-8-mq/src/sys/kern/vfs_vnops.c:594
Process 1495 (sshd) thread 0xffffff000c0cc720 (100114)
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff000cfe9ae8) locked _at_
/usr/src-8-mq/src/sys/kern/uipc_sockbuf.c:148
Process 1479 (sshd) thread 0xffffff000c13a000 (100103)
exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff001e2b8368) locked _at_
/usr/src-8-mq/src/sys/kern/uipc_sockbuf.c:148


--Artem