Panics and potential memory corruption when pulling out a uath device

From: Lucius Windschuh <lwindschuh_at_googlemail.com>
Date: Sun, 17 May 2009 22:54:56 +0200
With the newly imported uath driver, I was able to produce five
different panics.
Since four of them occur in unrelated kernel parts, this looks to me
like some kernel part is corrupting memory. But since I am not an
expert, here are backtraces for them:

First, the one which seems to be without memory corruption (minidump available):

panic: mtx_lock() of destroyed mutex _at_
/usr/src/sys/modules/wlan/../../net80211/ieee80211_node.c:1697

(kgdb) bt
#0  doadump () at pcpu.h:246
#1  0xc04949c9 in db_fncall (dummy1=-979506816, dummy2=0,
dummy3=-1068655593, dummy4=0xf3c47988 "_at_\231\235�001") at
/usr/src/sys/ddb/db_command.c:548
#2  0xc0494dc1 in db_command (last_cmdp=0xc0989c9c, cmd_table=0x0,
dopager=1) at /usr/src/sys/ddb/db_command.c:445
#3  0xc0494f1a in db_command_loop () at /usr/src/sys/ddb/db_command.c:498
#4  0xc0496d7d in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:229
#5  0xc06579d6 in kdb_trap (type=3, code=0, tf=0xf3c47b2c) at
/usr/src/sys/kern/subr_kdb.c:534
#6  0xc088bdce in trap (frame=0xf3c47b2c) at /usr/src/sys/i386/i386/trap.c:685
#7  0xc086f6fb in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#8  0xc0657b5a in kdb_enter (why=0xc08f8592 "panic", msg=0xc08f8592
"panic") at cpufunc.h:71
#9  0xc062a1a6 in panic (fmt=0xc08f6f47 "mtx_lock() of destroyed mutex
_at_ %s:%d") at /usr/src/sys/kern/kern_shutdown.c:559
#10 0xc061a925 in _mtx_lock_flags (m=0xc6af26b8, opts=0,
file=0xc858faff
"/usr/src/sys/modules/wlan/../../net80211/ieee80211_node.c",
line=1697) at /usr/src/sys/kern/kern_mutex.c:174
#11 0xc857445e in ieee80211_node_delucastkey (ni=0xc6af8000) at
/usr/src/sys/modules/wlan/../../net80211/ieee80211_node.c:1697
#12 0xc85760d6 in node_free (ni=0xc6af8000) at
/usr/src/sys/modules/wlan/../../net80211/ieee80211_node.c:999
#13 0xc8573992 in _ieee80211_free_node (ni=0xc6af8000) at
/usr/src/sys/modules/wlan/../../net80211/ieee80211_node.c:1622
#14 0xc84f5af0 in uath_bulk_tx_callback () from /boot/kernel/if_uath.ko
#15 0xc0594d27 in usb2_callback_wrapper (pq=0xc9448030) at
/usr/src/sys/dev/usb/usb_transfer.c:1962
#16 0xc0592716 in usb2_command_wrapper (pq=0xc9448030, xfer=0x0) at
/usr/src/sys/dev/usb/usb_transfer.c:2538
#17 0xc05927f8 in usb2_callback_proc (_pm=0xc9448044) at
/usr/src/sys/dev/usb/usb_transfer.c:1834
#18 0xc058febe in usb2_process (arg=0xc58d8ca4) at
/usr/src/sys/dev/usb/usb_process.c:139
#19 0xc06036e8 in fork_exit (callout=0xc058fde0 <usb2_process>,
arg=0xc58d8ca4, frame=0xf3c47d38) at /usr/src/sys/kern/kern_fork.c:830
#20 0xc086f7a0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:270

Now the strange faults:

2nd: (minidump available)
Fatal trap 12: page fault while in kernel mode
(kgdb) bt
#0  doadump () at pcpu.h:246
#1  0xc04949c9 in db_fncall (dummy1=-979506816, dummy2=0,
dummy3=-1068655593, dummy4=0xc4eb3a20 "_at_\231\235�001") at
/usr/src/sys/ddb/db_command.c:548
#2  0xc0494dc1 in db_command (last_cmdp=0xc0989c9c, cmd_table=0x0,
dopager=1) at /usr/src/sys/ddb/db_command.c:445
#3  0xc0494f1a in db_command_loop () at /usr/src/sys/ddb/db_command.c:498
#4  0xc0496d7d in db_trap (type=12, code=0) at /usr/src/sys/ddb/db_main.c:229
#5  0xc06579d6 in kdb_trap (type=12, code=0, tf=0xc4eb3c08) at
/usr/src/sys/kern/subr_kdb.c:534
#6  0xc088afcf in trap_fatal (frame=0xc4eb3c08, eva=3735929062) at
/usr/src/sys/i386/i386/trap.c:924
#7  0xc088b963 in trap (frame=0xc4eb3c08) at /usr/src/sys/i386/i386/trap.c:325
#8  0xc086f6fb in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#9  0xc063cad1 in softclock (arg=0xc09a4ea0) at
/usr/src/sys/kern/kern_timeout.c:335
#10 0xc0605975 in intr_event_execute_handlers (p=0xc516aa90,
ie=0xc51aa000) at /usr/src/sys/kern/kern_intr.c:1134
#11 0xc06065df in ithread_loop (arg=0xc50e7ca0) at
/usr/src/sys/kern/kern_intr.c:1147
#12 0xc06036e8 in fork_exit (callout=0xc0606540 <ithread_loop>,
arg=0xc50e7ca0, frame=0xc4eb3d38) at /usr/src/sys/kern/kern_fork.c:830
#13 0xc086f7a0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:270

3rd: (minidump available)
panic: Bad tailq NEXT(0xe59b4e40->tqh_last) != NULL
(kgdb) bt
#0  doadump () at pcpu.h:246
#1  0xc04949c9 in db_fncall (dummy1=1, dummy2=0, dummy3=-1061793024,
dummy4=0xc4eb39d8 "") at /usr/src/sys/ddb/db_command.c:548
#2  0xc0494dc1 in db_command (last_cmdp=0xc0989c9c, cmd_table=0x0,
dopager=1) at /usr/src/sys/ddb/db_command.c:445
#3  0xc0494f1a in db_command_loop () at /usr/src/sys/ddb/db_command.c:498
#4  0xc0496d7d in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:229
#5  0xc06579d6 in kdb_trap (type=3, code=0, tf=0xc4eb3b7c) at
/usr/src/sys/kern/subr_kdb.c:534
#6  0xc088bdce in trap (frame=0xc4eb3b7c) at /usr/src/sys/i386/i386/trap.c:685
#7  0xc086f6fb in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#8  0xc0657b5a in kdb_enter (why=0xc08f8592 "panic", msg=0xc08f8592
"panic") at cpufunc.h:71
#9  0xc062a1a6 in panic (fmt=0xc08c0c8d "Bad tailq NEXT(%p->tqh_last)
!= NULL") at /usr/src/sys/kern/kern_shutdown.c:559
#10 0xc063c780 in callout_reset_on (c=0xc09903a0, to_ticks=10,
ftn=0xc04d9c20 <dcons_timeout>, arg=0xc580ae00, cpu=0)
    at /usr/src/sys/kern/kern_timeout.c:626
#11 0xc04d9cf4 in dcons_timeout (v=0xc580ae00) at
/usr/src/sys/dev/dcons/dcons_os.c:241
#12 0xc063ccd4 in softclock (arg=0xc09a4ea0) at
/usr/src/sys/kern/kern_timeout.c:411
#13 0xc0605975 in intr_event_execute_handlers (p=0xc516aa90,
ie=0xc51aa000) at /usr/src/sys/kern/kern_intr.c:1134
#14 0xc06065df in ithread_loop (arg=0xc50e7ca0) at
/usr/src/sys/kern/kern_intr.c:1147
#15 0xc06036e8 in fork_exit (callout=0xc0606540 <ithread_loop>,
arg=0xc50e7ca0, frame=0xc4eb3d38) at /usr/src/sys/kern/kern_fork.c:830
#16 0xc086f7a0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:270

4th: (only textdump; PID 1368 is fsck_ufs)
panic: Bad link elm 0xc67e5f28 prev->next != elm
db:0:kdb.enter.panic>  bt
Tracing pid 1368 tid 100086 td 0xc67e5d80
kdb_enter(c09c58b4,c09c58b4,c09875f4,eae86b50,0,...) at kdb_enter+0x3a
panic(c09875f4,c67e5f28,100,c67e5d80,c67e5d80,...) at panic+0x136
_callout_stop_safe(c67e5f28,0,c09c9bf3,208,0,...) at _callout_stop_safe+0x391
sleepq_check_timeout(b,c06d2380,c67e5d80,0,100,...) at sleepq_check_timeout+0x73
sleepq_timedwait_sig(c0a7be84,5c,c09c6aa3,100,0,...) at
sleepq_timedwait_sig+0x21
_sleep(c0a7be84,0,15c,c09c6aa3,b,...) at _sleep+0x30e
kern_nanosleep(c67e5d80,eae86c64,eae86c6c,0,5dfc8c0,...) at kern_nanosleep+0xc1
nanosleep(c67e5d80,eae86cf8,8,c09cc50a,c0a2d800,...) at nanosleep+0x6f
syscall(eae86d38) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x281724ef, esp =
0xbfbfda1c, ebp = 0xbfbfda48 ---

5th: (only textdump; PID 11 is "intr")
panic: Bad link elm 0xc6f54568 next->prev != elm
db:0:kdb.enter.panic>  bt
Tracing pid 11 tid 100006 td 0xc6176480
kdb_enter(c09c58b4,c09c58b4,c09875d2,c5f3ec54,0,...) at kdb_enter+0x3a
panic(c09875d2,c6f54568,c09c6bbc,145,c0a7bef4,...) at panic+0x136
softclock(c0a7bec0,c5f3ecc8,c068cda4,c0a7fe00,c61b5c38,...) at softclock+0x10a
intr_event_execute_handlers(c6174a90,c61b5c00,c09c1671,4dd,c61b5c70,...)
at intr_event_execute_handlers+0x125
ithread_loop(c610fba0,c5f3ed38,c09c13ec,336,c6174a90,...) at ithread_loop+0x9f
fork_exit(c0679190,c610fba0,c5f3ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5f3ed70, ebp = 0 ---

The last two panics are from a differenct machine ("t400"), so I
exclude faulty memory.
The first three are from my machine "current".

Kernel config, etc: http://sites.google.com/site/lwfreebsd/Home/files/
Kernel version: CURRENT r192252

Any ideas?

Lucius
Received on Sun May 17 2009 - 18:54:57 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:47 UTC