Page faults from DEBUG_LOCKS (Re: panic: _sx_xlock (user map): xlock already held @ ../../../vm/vm_map.c:2997)

I think this one is caused by the DEBUG_LOCKS changes of jeffr and
Antoine a few weeks ago, which try to save stack traces when lockmgr
locks are acquired, but the stack tracing code sometimes page faults
when backtracing.

I think Antoine has an updated patch, but Jeffr hasn't yet committed
it.

Kris

On Thu, Aug 25, 2005 at 08:59:45PM +1000, Peter Jeremy wrote:
> I just got the same panic on -current from a couple of weeks ago (though
> newer than Kris's).  In my case, the system had been running happily
> for about 3 days and starting mplayer triggered it.  The saved message
> buffer was:
> 
> lock order reversal
> 1st 0xc4097224 vnode interlock (vnode interlock) _at_ /usr/src/sys/vm/vnode_pager.c:1181
> 2nd 0xc2007898 process lock (process lock) _at_ /usr/src/sys/i386/i386/trap.c:728
> KDB: stack backtrace:
> kdb_backtrace(c06d9b61,c2007898,c06d53c4,c06d53c4,c06efda6) at kdb_backtrace+0x2e
> witness_checkorder(c2007898,9,c06efda6,2d8,c0768ca0) at witness_checkorder+0x6c3
> _mtx_lock_flags(c2007898,0,c06efda6,2d8,c2046a80) at _mtx_lock_flags+0x8a
> trap_pfault(d961c9f8,0,3,d961c9f0,3) at trap_pfault+0x9f
> trap(8,28,28,c2046a80,d961ca68) at trap+0x3cd
> calltrap() at calltrap+0x5
> --- trap 0xc, eip = 0xc0688efd, esp = 0xd961ca38, ebp = 0xd961ca48 ---
> stack_save(d961ca68,0,c06d4042,a2,c2046a80) at stack_save+0x1d
> lockmgr(c40971b4,3041,c4097224,c2046a80,d961caf8) at lockmgr+0x5e
> vop_stdlock(d961cb4c,c4097224,9,c0724800,d961cb4c) at vop_stdlock+0x32
> VOP_LOCK_APV(c0724d40,d961cb4c,d961cb24,c06acac6,d961cb4c) at VOP_LOCK_APV+0xa6
> ffs_lock(d961cb4c,c073f760,d961cb34,3041,c409715c) at ffs_lock+0x19
> VOP_LOCK_APV(c0724800,d961cb4c,c06eaa10,d961cb50,c0552870) at VOP_LOCK_APV+0xa6
> vn_lock(c409715c,3041,c2046a80,c06eaa10,3041) at vn_lock+0xda
> vget(c409715c,3041,c2046a80,49e,0) at vget+0xbd
> vnode_pager_lock(c4057ad4,0,c06e8b6f,127,d961cc60) at vnode_pager_lock+0x18a
> vm_fault(c22eebb8,81ba000,1,0,c2046a80) at vm_fault+0x29d
> trap_pfault(d961cd38,1,81ba840,299,81ba840) at trap_pfault+0xf3
> trap(3b,3b,3b,0,ffffffff) at trap+0x260
> calltrap() at calltrap+0x5
> --- trap 0xc, eip = 0x81ba840, esp = 0xbfbfd08c, ebp = 0xffffffff ---
> panic: _sx_xlock (user map): xlock already held _at_ /usr/src/sys/vm/vm_map.c:2997
> KDB: stack backtrace:
> kdb_backtrace(c06d6223,c073a720,c06d66fd,d961c5e4,100) at kdb_backtrace+0x2e
> panic(c06d66fd,c06b48c7,c06e90a6,c06e9126,bb5) at panic+0xb7
> _sx_xlock(c22eebfc,c06e9126,bb5,c0548f04,d961c63c) at _sx_xlock+0x63
> _vm_map_lock_read(c22eebb8,c06e9126,bb5,161c6fc,0) at _vm_map_lock_read+0x4a
> vm_map_lookup(d961c6e0,0,1,d961c6e4,d961c6d4) at vm_map_lookup+0x2e
> vm_fault(c22eebb8,0,1,0,c2046a80) at vm_fault+0x7e
> trap_pfault(d961c7ac,0,3,0,3) at trap_pfault+0xf3
> trap(8,28,28,4,3ee) at trap+0x3cd
> calltrap() at calltrap+0x5
> --- trap 0xc, eip = 0xc0688290, esp = 0xd961c7ec, ebp = 0xd961c828 ---
> db_read_bytes(3,3,d961c83c,d961c864,c0452298) at db_read_bytes+0x30
> db_get_value(3,4,0,d961c8ec,c0688d2b) at db_get_value+0x22
> db_numargs(ffffffff,d961c890,d961c8a0,d961cd38,c0698b30) at db_numargs+0x24
> db_backtrace(c2046a80,0,ffffffff,81ba840,ffffffff) at db_backtrace+0x1eb
> db_trace_self(c06d81b9,d961c960,c0553093,c06d9b61,c2007898) at db_trace_self+0x4d
> kdb_backtrace(c06d9b61,c2007898,c06d53c4,c06d53c4,c06efda6) at kdb_backtrace+0x2e
> witness_checkorder(c2007898,9,c06efda6,2d8,c0768ca0) at witness_checkorder+0x6c3
> _mtx_lock_flags(c2007898,0,c06efda6,2d8,c2046a80) at _mtx_lock_flags+0x8a
> trap_pfault(d961c9f8,0,3,d961c9f0,3) at trap_pfault+0x9f
> trap(8,28,28,c2046a80,d961ca68) at trap+0x3cd
> calltrap() at calltrap+0x5
> --- trap 0xc, eip = 0xc0688efd, esp = 0xd961ca38, ebp = 0xd961ca48 ---
> stack_save(d961ca68,0,c06d4042,a2,c2046a80) at stack_save+0x1d
> lockmgr(c40971b4,3041,c4097224,c2046a80,d961caf8) at lockmgr+0x5e
> vop_stdlock(d961cb4c,c4097224,9,c0724800,d961cb4c) at vop_stdlock+0x32
> VOP_LOCK_APV(c0724d40,d961cb4c,d961cb24,c06acac6,d961cb4c) at VOP_LOCK_APV+0xa6
> ffs_lock(d961cb4c,c073f760,d961cb34,3041,c409715c) at ffs_lock+0x19
> VOP_LOCK_APV(c0724800,d961cb4c,c06eaa10,d961cb50,c0552870) at VOP_LOCK_APV+0xa6
> vn_lock(c409715c,3041,c2046a80,c06eaa10,3041) at vn_lock+0xda
> vget(c409715c,3041,c2046a80,49e,0) at vget+0xbd
> vnode_pager_lock(c4057ad4,0,c06e8b6f,127,d961cc60) at vnode_pager_lock+0x18a
> vm_fault(c22eebb8,81ba000,1,0,c2046a80) at vm_fault+0x29d
> trap_pfault(d961cd38,1,81ba840,299,81ba840) at trap_pfault+0xf3
> trap(3b,3b,3b,0,ffffffff) at trap+0x260
> calltrap() at calltrap+0x5
> --- trap 0xc, eip = 0x81ba840, esp = 0xbfbfd08c, ebp = 0xffffffff ---
> panic: _mtx_lock_sleep: recursed on non-recursive mutex lockbuilder mtxpool _at_ /usr/src/sys/kern/kern_sx.c:157
> 
> KDB: enter: panic
> 
> then a pile of
> Fatal trap 3: breakpoint instruction fault while in kernel mode
> until it reset.
> 
> -- 
> Peter Jeremy
>