Re: Double mutex destruction

In the last episode (Aug 17), Alexander Kabaev said:
> I left development box running 7.0 sitting idle at work overnight and
> was greeted with the following panic this morning:
> 
> panic: mtx_lock() of destroyed mutex _at_ ../../../net/route.c:1303
> cpuid = 1
> KDB: enter: panic
> [thread pid 4088 tid 100128 ]
> Stopped at      kdb_enter+0x32: leave
> db> bt
> Tracing pid 4088 tid 100128 td 0xc5539cc0
> kdb_enter(c07c56bd,1,c07c4619,f199f9bc,1,...) at kdb_enter+0x32
> panic(c07c4619,c07d2ce5,517,507,c5386a7c,...) at panic+0x124
> _mtx_lock_flags(c5aeb510,0,c07d2ce5,517,f199fa28,...) at _mtx_lock_flags+0x65
> rt_check(f199fa20,f199fa3c,c541ec30,c080a6c4,c07d2cee,...) at rt_check+0x111
> arpresolve(c51e4000,c5584bb8,c5587d00,c541ec30,f199fa56,...) at arpresolve+0xb0
[...]

I get this about once a week or so, and Andre Guibert de Bruet
<andy_at_siliconlandmark.com> has also reported it.  Occasionally instead
of an assertion panic, I get a trap 12:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 03
fault virtual address	= 0x188
fault code		= supervisor read, page not present
instruction pointer	= 0x20:0xc05dc855
stack pointer	        = 0x28:0xe74b0920
frame pointer	        = 0x28:0xe74b0938
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= resume, IOPL = 0
current process		= 2623 (epic-EPIC4-2.6)
trap number		= 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper(c082afcc,e74b07fc,c05e9541,c08435d9,0,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08435d9,0,c080f97f,e74b0808,0,...) at kdb_backtrace+0x29
panic(c080f97f,c084484e,c83e377c,1,1,...) at panic+0x111
trap_fatal(c0844750,c,c082707b,efa498f6,c83e3558,...) at trap_fatal+0x383
trap(e74b08e0) at trap+0x11b
calltrap() at calltrap+0x6
--- trap 0xc, eip = 0xc05dc855, esp = 0xe74b0920, ebp = 0xe74b0938 ---
_mtx_lock_sleep(c62d3d80,c83e6000,0,0,0,...) at _mtx_lock_sleep+0x85
rt_check(e74b0980,e74b099c,c5a64e10,0,0,...) at rt_check+0x120
arpresolve(c3bd0c00,c62d4d20,c3fff400,c5a64e10,e74b09b6,...) at arpresolve+0xb4
ether_output(c3bd0c00,c3fff400,c5a64e10,c62d4d20,c807d7e0,...) at ether_output+0x8e
ip_output(c3fff400,0,e74b0a28,0,0,...) at ip_output+0xb45
tcp_output(c744a000,c437a100,e74b0c60,1,0,...) at tcp_output+0x11de
tcp_usr_send(c7fc3630,0,c437a100,0,0,...) at tcp_usr_send+0x262
sosend_generic(c7fc3630,0,e74b0c60,c437a100,0,...) at sosend_generic+0x6a5
sosend(c7fc3630,0,e74b0c60,0,0,...) at sosend+0x3f
soo_write(c9869948,e74b0c60,c9871000,0,c83e6000,...) at soo_write+0x4b
dofilewrite(e74b0c60,ffffffff,ffffffff,0,c9869948,...) at dofilewrite+0x97
kern_writev(c83e6000,3,e74b0c60,bfbf8d02,0,...) at kern_writev+0x58
write(c83e6000,e74b0cfc,c,16,e74b0d2c,...) at write+0x4f
syscall(e74b0d38) at syscall+0x365
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (4, FreeBSD ELF32, write), eip = 0x28448577, esp = 0xbfbf8c8c, ebp = 0xbfbf8cb8 ---
Uptime: 6h40m30s
Physical memory: 1015 MB
Dumping 246 MB: 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7

#0  doadump () at pcpu.h:195
195	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) (kgdb) #0  doadump () at pcpu.h:195
#1  0xc05e9287 in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2  0xc05e9570 in panic (fmt=Variable "fmt" is not available.) at ../../../kern/kern_shutdown.c:563
#3  0xc07cabd3 in trap_fatal (frame=0xe74b08e0, eva=392) at ../../../i386/i386/trap.c:872
#4  0xc07cb5cb in trap (frame=0xe74b08e0) at ../../../i386/i386/trap.c:277
#5  0xc07b070b in calltrap () at ../../../i386/i386/exception.s:139
#6  0xc05dc855 in _mtx_lock_sleep (m=0xc62d3d80, tid=3359531008, opts=0, file=0x0, line=0) at ../../../kern/kern_mutex.c:360
#7  0xc0695a60 in rt_check (lrt=0xe74b0980, lrt0=0xe74b099c, dst=0xc5a64e10) at ../../../net/route.c:1303
#8  0xc069f644 in arpresolve (ifp=0xc3bd0c00, rt0=0xc62d4d20, m=0xc3fff400, dst=0xc5a64e10, desten=0xe74b09b6 " ├D⌠ ├") at ../../../netinet/if_ether.c:373
#9  0xc068964e in ether_output (ifp=0xc3bd0c00, m=0xc3fff400, dst=0xc5a64e10, rt0=0xc62d4d20) at ../../../net/if_ethersubr.c:175
#10 0xc06ba9a5 in ip_output (m=0xc3fff400, opt=0x0, ro=0xe74b0a28, flags=Variable "flags" is not available.) at ../../../netinet/ip_output.c:547
#11 0xc06c2c7e in tcp_output (tp=0xc744a000) at ../../../netinet/tcp_output.c:1125
#12 0xc06cba12 in tcp_usr_send (so=0xc7fc3630, flags=Variable "flags" is not available.) at ../../../netinet/tcp_usrreq.c:839
#13 0xc06406a5 in sosend_generic (so=0xc7fc3630, addr=0x0, uio=0xe74b0c60, top=0xc437a100, control=0x0, flags=0, td=0xc83e6000) at ../../../kern/uipc_socket.c:1241
#14 0xc063c34f in sosend (so=0xc7fc3630, addr=0x0, uio=0xe74b0c60, top=0x0, control=0x0, flags=0, td=0xc83e6000) at ../../../kern/uipc_socket.c:1287
#15 0xc0624a3b in soo_write (fp=0xc9869948, uio=0xe74b0c60, active_cred=0xc9871000, flags=0, td=0xc83e6000) at ../../../kern/sys_socket.c:104
#16 0xc061e0c7 in dofilewrite (td=0xc83e6000, fd=3, fp=0xc9869948, auio=0xe74b0c60, offset=-1, flags=0) at file.h:254
#17 0xc061e3b8 in kern_writev (td=0xc83e6000, fd=3, auio=0xe74b0c60) at ../../../kern/sys_generic.c:404
#18 0xc061e42f in write (td=0xc83e6000, uap=0xe74b0cfc) at ../../../kern/sys_generic.c:320
#19 0xc07cb1e5 in syscall (frame=0xe74b0d38) at ../../../i386/i386/trap.c:1008
#20 0xc07b0770 in Xint0x80_syscall () at ../../../i386/i386/exception.s:196
#21 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

-- 
	Dan Nelson
	dnelson_at_allantgroup.com