ZFS and deadlock with {nullfs,NFS}

From: Joe Marcus Clarke <marcus_at_FreeBSD.org>
Date: Wed, 20 Jun 2007 11:53:43 -0400
I've resurrected by amd64 Tinderbox with a ZFS base, and I've been
seeing a 100% reproducible deadlock when I use it with either localhost
NFS or nullfs.  When this occurs, the CPU is 100% idle, but I can no
longer connect via SSH, and the box will only reboot from the debugger.
I know there are some tuning bits I can tweak, but all I've run across
is for memory consumption.  Any pointers would be helpful.  I'm also at
the debugger, so if there is anything I can do to help troubleshoot why
this is happening, please let me know.  

This box is -CURRENT as of June 19, 2007.  It has a GENERIC kernel minus
devices I do not have (i.e. SMP kernel).  I am currently using nullfs
for the Tinderbox.  The process that most regularly locks up is mtree.
Here is the trace:

FreeBSD cobbler.marcuscom.com 7.0-CURRENT FreeBSD 7.0-CURRENT #32: Tue
Jun 19 12:53:28 EDT 2007
marcus_at_cobbler.marcuscom.com:/usr/obj/usr/src/sys/COBBLER  amd64

Tracing command mtree pid 34844 tid 100169 td 0xffffff0060350680
sched_switch() at sched_switch+0x176
mi_switch() at mi_switch+0x231
sleepq_switch() at sleepq_switch+0xc7
sleepq_wait() at sleepq_wait+0x44
_cv_wait() at _cv_wait+0x14e
zio_wait() at zio_wait+0x5f
dbuf_read() at dbuf_read+0x2d5
dbuf_findbp() at dbuf_findbp+0xf7
dbuf_prefetch() at dbuf_prefetch+0xa9
dmu_zfetch_dofetch() at dmu_zfetch_dofetch+0xd4
dmu_zfetch() at dmu_zfetch+0x603
dbuf_read() at dbuf_read+0x535
dnode_hold_impl() at dnode_hold_impl+0xe7
dmu_bonus_hold() at dmu_bonus_hold+0x32
zfs_zget() at zfs_zget+0x6b
zfs_dirent_lock() at zfs_dirent_lock+0x2fc
zfs_dirlook() at zfs_dirlook+0x58
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x13d
vfs_cache_lookup() at vfs_cache_lookup+0xf8
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x6d
lookup() at lookup+0x363
namei() at namei+0x2ca
kern_lstat() at kern_lstat+0x5e
lstat() at lstat+0x2a
syscall() at syscall+0x1ce
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x800807e5c, rsp =
0x7fffffffe5b8, rbp = 0x556170 ---

A full process list from the debugger can be found at
http://www.marcuscom.com/downloads/cobbler_proc.txt .

Joe

-- 
Joe Marcus Clarke
FreeBSD GNOME Team      ::      gnome_at_FreeBSD.org
FreeNode / #freebsd-gnome
http://www.FreeBSD.org/gnome

Received on Wed Jun 20 2007 - 13:53:50 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:12 UTC