Re: DEADLKRES crash

On 2014-08-18 15:45, Ryan Stone wrote:
> On Mon, Aug 18, 2014 at 11:21 AM, Larry Rosenman <ler_at_lerctr.org> 
> wrote:
>> I got the following:
>> 
>> borg.lerctr.org dumped core - see /var/crash/vmcore.8
>> 
>> Mon Aug 18 07:30:42 CDT 2014
>> 
>> FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #63 
>> r269784M: Sun Aug 10 12:33:07 CDT 2014     
>> root_at_borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER  amd64
>> 
>> panic: deadlkres: possible deadlock detected for 0xfffff8002abeb000, 
>> blocked for 1800926 ticks
>> 
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and 
>> you are
>> welcome to change it and/or distribute copies of it under certain 
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for 
>> details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>> 
>> Unread portion of the kernel message buffer:
>> panic: deadlkres: possible deadlock detected for 0xfffff8002abeb000, 
>> blocked for 1800926 ticks
>> 
>> cpuid = 3
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
>> 0xfffffe100bff1a10
>> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe100bff1ac0
>> vpanic() at vpanic+0x126/frame 0xfffffe100bff1b00
>> panic() at panic+0x43/frame 0xfffffe100bff1b60
>> deadlkres() at deadlkres+0x35c/frame 0xfffffe100bff1bb0
>> fork_exit() at fork_exit+0x84/frame 0xfffffe100bff1bf0
>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe100bff1bf0
>> --- trap 0, rip = 0, rsp = 0xfffffe100bff1cb0, rbp = 0 ---
>> Uptime: 7d14h14m38s
> 
> The first thing that I'd like to see is (in kgdb):
> 
> set $td=(struct thread)0xfffff8002abeb000
> tid $td->td_tid
> bt
> 
> That will show us the backtrace of the thread that was blocked for so 
> long.
0  doadump (textdump=1) at pcpu.h:219
219	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) set $td=(struct thread)0xfffff8002abeb000
Invalid cast.
(kgdb) set $td=(struct thread*)0xfffff8002abeb000
Current language:  auto; currently minimal
(kgdb) tid $td->td_tid
[Switching to thread 469 (Thread 100681)]#0  sched_switch (
     td=0xfffff8002abeb000, newtd=<value optimized out>,
     flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1931
1931			cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0  sched_switch (td=0xfffff8002abeb000, newtd=<value optimized out>,
     flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1931
#1  0xffffffff80a107d9 in mi_switch (flags=260, newtd=0x0)
     at /usr/src/sys/kern/kern_synch.c:493
#2  0xffffffff80a4c442 in sleepq_switch (wchan=<value optimized out>,
     pri=<value optimized out>) at 
/usr/src/sys/kern/subr_sleepqueue.c:552
#3  0xffffffff80a4c2a3 in sleepq_wait (wchan=0xfffff80070a4dd50, pri=96)
     at /usr/src/sys/kern/subr_sleepqueue.c:631
#4  0xffffffff809eb1fa in sleeplk (lk=<value optimized out>,
     flags=<value optimized out>, ilk=<value optimized out>,
     wmesg=<value optimized out>, pri=<value optimized out>,
     timo=<value optimized out>) at /usr/src/sys/kern/kern_lock.c:225
#5  0xffffffff809eaa06 in __lockmgr_args (lk=0xfffff80070a4dd50,
     flags=<value optimized out>, ilk=0xfffff80070a4dd80,
     wmesg=<value optimized out>, pri=<value optimized out>,
     timo=<value optimized out>) at /usr/src/sys/kern/kern_lock.c:931
#6  0xffffffff8092e092 in nfs_lock1 (ap=<value optimized out>) at 
lockmgr.h:97
#7  0xffffffff80f2d57c in VOP_LOCK1_APV (vop=<value optimized out>,
     a=<value optimized out>) at vnode_if.c:2082
#8  0xffffffff80abd22a in _vn_lock (vp=0xfffff80070a4dce8,
     flags=<value optimized out>,
     file=0xffffffff8110db88 "/usr/src/sys/kern/vfs_subr.c", line=2137)
     at vnode_if.h:859
#9  0xffffffff80aad4e7 in vget (vp=0xfffff80070a4dce8, flags=524544,
---Type <return> to continue, or q <return> to quit---
     td=0xfffff8002abeb000) at /usr/src/sys/kern/vfs_subr.c:2137
#10 0xffffffff80aa1491 in vfs_hash_get (mp=0xfffff8002aa1e990,
     hash=1741450670, flags=<value optimized out>, td=0xfffff8002abeb000,
     vpp=0xfffffe100c75c670, fn=0xffffffff80935820 <newnfs_vncmpf>)
     at /usr/src/sys/kern/vfs_hash.c:88
#11 0xffffffff809314bd in ncl_nget (mntp=0xfffff8002aa1e990,
     fhp=0xfffff80070ccf4a4 "\001", fhsize=12, npp=0xfffffe100c75c6e0,
     lkflags=<value optimized out>)
     at /usr/src/sys/fs/nfsclient/nfs_clnode.c:114
#12 0xffffffff809340fd in nfs_statfs (mp=0xfffff8002aa1e990,
     sbp=0xfffff8002aa1ea48) at 
/usr/src/sys/fs/nfsclient/nfs_clvfsops.c:288
#13 0xffffffff80aa7ade in __vfs_statfs (mp=0x0, sbp=0xfffff8002aa1ea48)
     at /usr/src/sys/kern/vfs_mount.c:1706
#14 0xffffffff80ab4f5e in kern_getfsstat (td=0xfffff8002abeb000,
     buf=<value optimized out>, bufsize=<value optimized out>,
     bufseg=UIO_USERSPACE, flags=<value optimized out>)
     at /usr/src/sys/kern/vfs_syscalls.c:511
#15 0xffffffff80e1625a in amd64_syscall (td=0xfffff8002abeb000, 
traced=0)
     at subr_syscall.c:133
#16 0xffffffff80df760b in Xfast_syscall ()
     at /usr/src/sys/amd64/amd64/exception.S:390
#17 0x00000008010fc83a in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c)     E-Mail: ler_at_lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688