Re: panic: mtx_lock_spin: recursed on non-recursive mutex uart_hwmtx...

On Mon, Jan 20, 2014 at 11:37 AM, Konstantin Belousov
<kostikbel_at_gmail.com>wrote:

> On Mon, Jan 20, 2014 at 07:36:15AM -0800, David Wolfskill wrote:
> > Jan 20 07:02:16 freebeast syslogd: exiting on signal 15
> > Waiting (max 60 seconds) for system process `vnlru' to stop...done
> > Waiting (max 60 seconds) for system process `syncer' to stop...
> > Syncing disks, vnodes remaining...2 4 4 4 3 3 3 2 1 1 0 0 0 0 done
> > Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
> > All buffers synced.
> > lock order reversal:
> >  1st 0xc7336c68 ufs (ufs) _at_ /usr/src/sys/kern/vfs_mount.c:1237
> >  2nd 0xc76846dc devfs (devfs) _at_ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1412
> > KDB: stack backtrace:
> > db_trace_self_wrapper(c112d57c,a3231,e1fb9888,c0b1bfe7,1,...) at
> db_trace_self_wrapper+0x2d/frame 0xe1fb9828
> > kdb_backtrace(c1131131,c76846dc,c1124024,c6dad2d0,c1162dc1,...) at
> kdb_backtrace+0x30/frame 0xe1fb9890
> > witness_checkorder(c76846dc,9,c1162dc1,584,0,...) at
> witness_checkorder+0xc8a/frame 0xe1fb98e0
> > __lockmgr_args(c76846dc,80400,c76846fc,0,0,0,c1162dc1,584) at
> __lockmgr_args+0x844/frame 0xe1fb99b4
> > vop_stdlock(e1fb9a28,c6dad3a0,c6dad408,c169bc38,e1fb9a4c,...) at
> vop_stdlock+0x4d/frame 0xe1fb99e4
> > VOP_LOCK1_APV(c13828dc,e1fb9a28,0,0,c13cded0,...) at
> VOP_LOCK1_APV+0x104/frame 0xe1fb9a10
> > _vn_lock(c76846a8,80400,c1162dc1,584,c1393aac,...) at
> _vn_lock+0xa1/frame 0xe1fb9a50
> > ffs_flushfiles(c753f000,a,c6f0a310,e1fb9b38,c0d416e3,...) at
> ffs_flushfiles+0x14c/frame 0xe1fb9a9c
> > softdep_flushfiles(c753f000,2,c6f0a310,0,0,...) at
> softdep_flushfiles+0x6e/frame 0xe1fb9af0
> > ffs_unmount(c753f000,80000,e1fb9b70,518,c6f0a310,...) at
> ffs_unmount+0x194/frame 0xe1fb9b38
> > dounmount(c753f000,80000,c6f0a310,0,e1db3eb0,...) at
> dounmount+0x4dc/frame 0xe1fb9b98
> > vfs_unmountall(e1db4058,0,c112895a,144,c6d8b1a0,...) at
> vfs_unmountall+0x4e/frame 0xe1fb9bb8
> > kern_reboot(4008,0,c112895a,c3,0,...) at kern_reboot+0x5af/frame
> 0xe1fb9c20
> > sys_reboot(c6f0a310,e1fb9cc8,14,c112b5b7,7a2,...) at
> sys_reboot+0x6f/frame 0xe1fb9c40
> > syscall(e1fb9d08) at syscall+0x2de/frame 0xe1fb9cfc
> > Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe1fb9cfc
> > --- syscall (55, FreeBSD ELF32, sys_reboot), eip = 0x805a9df, esp =
> 0xbfbfd88c, ebp = 0xbfbfd960 ---
> > Uptime: 4m28s
> > panic: mtx_lock_spin: recursed on non-recursive mutex uart_hwmtx _at_
> /usr/src/sys/dev/uart/uart_cpu.h:94
> >
> > cpuid = 0
> > KDB: stack backtrace:
> > db_trace_self_wrapper(c112d57c,2f727375,2f637273,2f737973,2f766564,...)
> at db_trace_self_wrapper+0x2d/frame 0xe1fb9a60
> > kdb_backtrace(c12ec1f3,0,c11269a2,e1fb9b38,e1fb9b08,...) at
> kdb_backtrace+0x30/frame 0xe1fb9ac8
> > vpanic(c140d0c8,100,c11269a2,e1fb9b38,e1fb9b38,...) at
> vpanic+0x11f/frame 0xe1fb9b08
> > kassert_panic(c11269a2,c10f66d8,c10f6758,5e,8,...) at
> kassert_panic+0xea/frame 0xe1fb9b2c
> > __mtx_lock_spin_flags(c7291830,0,c10f6758,5e,c137d684,...) at
> __mtx_lock_spin_flags+0x190/frame 0xe1fb9b5c
> > ns8250_bus_grab(c7291800,c7062b28,c137d684,5e,c140b9a4,...) at
> ns8250_bus_grab+0x40/frame 0xe1fb9b78
> > uart_grab(c13fd070) at uart_grab+0x83/frame 0xe1fb9b9c
> > uart_cngrab(c137d69c,88888889,e1fb9c20,c0ac7d43,c1128e43,...) at
> uart_cngrab+0x12/frame 0xe1fb9ba8
> > cngrab(c1128e43,1c,c112895a,144,c6d8b1a0,...) at cngrab+0x33/frame
> 0xe1fb9bb8
> > kern_reboot(4008,0,c112895a,c3,0,...) at kern_reboot+0x6d3/frame
> 0xe1fb9c20
> > sys_reboot(c6f0a310,e1fb9cc8,14,c112b5b7,7a2,...) at
> sys_reboot+0x6f/frame 0xe1fb9c40
> > syscall(e1fb9d08) at syscall+0x2de/frame 0xe1fb9cfc
> > Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe1fb9cfc
> > --- syscall (55, FreeBSD ELF32, sys_reboot), eip = 0x805a9df, esp =
> 0xbfbfd88c, ebp = 0xbfbfd960 ---
> > KDB: enter: panic
> > [ thread pid 1 tid 100002 ]
> > Stopped at      kdb_enter+0x3d: movl    $0,kdb_why
> > db> show locks
> > exclusive sleep mutex Giant (Giant) r = 0 (0xc1724910) locked _at_
> /usr/src/sys/vm/swap_pager.c:2365
> > db>
> >
> >
> > When I issued "reset", the machine came back up normally (in slice
> > 1 -- stable/9).  I then rebooted to slice 4 (head), and issued the
> > same commands I usually do to set the default boot slice back to 1, then
> > "shutdown -p now" -- and re-created the panic.
> >
> > I can leave the machine up for a while if anyone has suggestions for me
> > to poke around.  I have a local private mirror of the FreeBSD SVN repos
> > and a spare bootable slice; I'm williing to try patches.  The machine
> > isn't especially fast, but it's generally OK.
> >
> > If it would be worthwhile, I could reboot my laptop to slice 4 (head) &
> > attempt the same reset-to-slice-1-default & power-off.
> >
> > This definitely did not happen _at_r260875, and it appears to be quite
> > repeatable _at_r260904
>
> You do not have option WITNESS_SKIPSPIN in your kernel config, do you ?
>

My recent source upgrade from 10.0-RELEASE r260689 to 11.0-CURRENT r260850
left me with a GENERIC kernel with the following defined:

options WITNESS # Enable checks to detect deadlocks and cycles
options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed

why?

-Tom