ZFS : panic("sleeping thread")

Hi,

While recent ZFS improvements got rid of random hangs I used to see,
there's still one problem that I keep running into -- panic in ZFS
under heavy load. I can reproduce it by doing a build with -j16 in a
jail running i386 binaries on -CURRENT/amd64 running on a box with
quad-core CPU. It takes a while to reproduce, but it usually shows up
within couple of hours.

Sleeping thread (tid 100606, pid 32147) owns a non-sleepable lock
sched_switch() at sched_switch+0xed
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sx_xlock_hard() at _sx_xlock_hard+0x1f0
_sx_xlock() at _sx_xlock+0x4e
rrw_exit() at rrw_exit+0x1d
zfs_freebsd_getattr() at zfs_freebsd_getattr+0x2be
VOP_GETATTR_APV() at VOP_GETATTR_APV+0x44
filt_vfsread() at filt_vfsread+0x51
knote() at knote+0xc2
VOP_WRITE_APV() at VOP_WRITE_APV+0x11f
vn_write() at vn_write+0x279
dofilewrite() at dofilewrite+0x85
kern_writev() at kern_writev+0x60
write() at write+0x54
ia32_syscall() at ia32_syscall+0x236
Xint0x80_syscall() at Xint0x80_syscall+0x85
--- syscall (4, FreeBSD ELF32, write), rip = 0x78162153, rsp =
0xffff945c, rbp = 0xffff9478 ---

It appears that locking within ZFS conflicts with vnode locking. The
back-trace is always the same.

For now, I've applied following patch to disable the panic, but it
would be good if someone familiar with VFS locking in FreeBSD could
take a look.
If you need any additional info, let me know.

Thanks,
--Artem

diff -r 930d975c8103 src/sys/kern/subr_turnstile.c
--- a/sys/kern/subr_turnstile.c	Fri Dec 05 16:12:43 2008 -0800
+++ b/sys/kern/subr_turnstile.c	Fri Dec 12 14:31:16 2008 -0800
_at__at_ -219,7 +219,10 _at__at_
 #ifdef DDB
 			db_trace_thread(td, -1);
 #endif
-			panic("sleeping thread");
+                        /* Don't propagate priority to a sleeping thread. */
+			thread_unlock(td);
+			return;
+			// panic("sleeping thread");
 		}

 		/*