4 LORs and freeze with wi(4)

From: Jon Noack <noackjr_at_alumni.rice.edu>
Date: Sat, 26 Mar 2005 17:10:31 -0600
I'm trying to get my Compaq Presario 1800T laptop up and running with 
-CURRENT.  I successfully installed the March snapshot, cvsupped (today 
_at_ ~1:30am CST), and built a new world/kernel.  Attempting to copy over 
some ports distfiles from another machine via scp causes the machine to 
freeze (console not responsive and machine not pingable).  I am able to 
break into ddb, but further testing shows that any heavy network traffic 
will trigger the freeze.

I had no problems with the snapshot, but I did notice the machine was 
spending ~65% of the CPU servicing the cbb0 interrupt under heavy 
network traffic.  The wi(4) card is a Linksys Instant Wireless WPC11 3.0 
(802.11b only), so heavy traffic is ~500 KB/sec and ~600 interrupts/sec. 
  Why does 600 interrupts/sec use 65% on a 900 MHz Pentium 3?  I guess 
witness could be causing that, but 1 ms and close to 1 million clock 
cycles per interrupt?  Either I don't understand what is going on here 
(quite possible) or something is very wrong.

I have a core and can easily reproduce this problem.  Please let me know 
how I need to proceed from here to help resolve this.  I wasn't sure 
exactly what info to extract from ddb or kgdb, so I hope what I did get 
was helpful.  Any pointers on how to get more useful information are 
appreciated.

Jon

LORs (the first similar to 
http://sources.zabbadoz.net/freebsd/lor.html#035):
**********************************************************************
lock order reversal
  1st 0xc1428d4c wi0 (network driver) _at_ /usr/src/sys/dev/wi/if_wi.c:601
  2nd 0xc139707c radix node head (radix node head) _at_ 
/usr/src/sys/net/route.c:148
KDB: stack backtrace:
witness_checkorder(c139707c,9,c0651b0f,94) at witness_checkorder+0x4d6
_mtx_lock_flags(c139707c,0,c0651b0f,94,7) at _mtx_lock_flags+0x5f
rtalloc1(cdf0baec,1,0,210,b01a8c0) at rtalloc1+0x65
arplookup(0,80246,cdf0bb34,c04ce49d,138) at arplookup+0x3a
arpintr(c13fa700,c13fa700,0,c13fa700,3c) at arpintr+0x303
ether_demux(c1428000,c13fa700,806,c1428000,c1399000) at ether_demux+0x228
ether_input(c1428000,c13fa700,c067fa20,436,0) at ether_input+0x19a
ieee80211_input(c1428254,c13fa700,c1399000,2f,2ec039) at 
ieee80211_input+0xd39
wi_intr(c1428000,0,0,c1223be8,0) at wi_intr+0x72d
ithread_loop(c121c700,cdf0bd48,28,c121c700,c04c4b76) at ithread_loop+0x113
fork_exit(c04c4b76,c121c700,cdf0bd48) at fork_exit+0x9b
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xcdf0bd7c, ebp = 0 ---

lock order reversal
  1st 0xc14c5924 rtentry (rtentry) _at_ /usr/src/sys/netinet/if_ether.c:445
  2nd 0xc1428d4c wi0 (network driver) _at_ /usr/src/sys/dev/wi/if_wi.c:876
KDB: stack backtrace:
witness_checkorder(c1428d4c,9,c0640083,36c) at witness_checkorder+0x4d6
_mtx_lock_flags(c1428d4c,0,c0640083,36c,c064aef7) at _mtx_lock_flags+0x5f
wi_start(c1428000,202,0,c13fa1d6,0) at wi_start+0x34
ether_output_frame(c1428000,c13fa100,ffffffff,ffffffff,c05dffff) at 
ether_output_frame+0x1b5
ether_output(c1428000,c13fa100,d2d89a64,0,c14744c8) at ether_output+0x34b
arprequest(d2d89b3c) at arprequest+0xda
arpresolve(c1428000,c14c58c4,c13fa200,d2d89b38,d2d89ac0) at arpresolve+0x259
ether_output(c1428000,c13fa200,d2d89b38,c14c58c4) at ether_output+0x82
ip_output(c13fa200,0,d2d89b34,1,0,c14c37bc) at ip_output+0x844
udp_send(c14c167c,0,c13fa200,0,0,c140f730) at udp_send+0x46a
sosend(c14c167c,0,d2d89c48,c13fa200,0,0,c140f730) at sosend+0x657
kern_sendit(c140f730,4,d2d89cc8,0,0) at kern_sendit+0xf5
sendit(d2d89cc8,0,807a021,0,0) at sendit+0x14f
sendto(c140f730,d2d89d14,bfbfddd0,6,3) at sendto+0x4d
syscall(2f,2f,2f,bfbfe95c,2812b624) at syscall+0x1f2
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (133, FreeBSD ELF32, sendto), eip = 0x280bf6ab, esp = 
0xbfbfddcc, ebp = 0xbfbfddec ---

lock order reversal
  1st 0xc14c3798 inp (udpinp) _at_ /usr/src/sys/netinet/udp_usrreq.c:772
  2nd 0xc1428d4c wi0 (network driver) _at_ /usr/src/sys/dev/wi/if_wi.c:876
KDB: stack backtrace:
witness_checkorder(c1428d4c,9,c0640083,36c) at witness_checkorder+0x4d6
_mtx_lock_flags(c1428d4c,0,c0640083,36c,c064aef7) at _mtx_lock_flags+0x5f
wi_start(c1428000,2,0,c13f9cb2,0) at wi_start+0x34
ether_output_frame(c1428000,c13f9c00,0,de90700,2c67) at 
ether_output_frame+0x1b5
ether_output(c1428000,c13f9c00,d2d89b38,c14c58c4) at ether_output+0x34b
ip_output(c13f9c00,0,d2d89b34,1,0,c14c3708) at ip_output+0x844
udp_send(c14c17c8,0,c13f9c00,0,0,c140f730) at udp_send+0x46a
sosend(c14c17c8,0,d2d89c48,c13f9c00,0,0,c140f730) at sosend+0x657
kern_sendit(c140f730,4,d2d89cc8,0,0) at kern_sendit+0xf5
sendit(d2d89cc8,0,807a021,0,0) at sendit+0x14f
sendto(c140f730,d2d89d14,bfbfddd0,6,b) at sendto+0x4d
syscall(2f,2f,2f,bfbfe93c,2812b624) at syscall+0x1f2
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (133, FreeBSD ELF32, sendto), eip = 0x280bf6ab, esp = 
0xbfbfddcc, ebp = 0xbfbfddec ---

lock order reversal
  1st 0xc1428d4c wi0 (network driver) _at_ /usr/src/sys/dev/wi/if_wi.c:601
  2nd 0xc06f512c tcp (tcp) _at_ /usr/src/sys/netinet/tcp_input.c:614
KDB: stack backtrace:
witness_checkorder(c06f512c,9,c0655279,266) at witness_checkorder+0x4d6
_mtx_lock_flags(c06f512c,0,c0655279,266) at _mtx_lock_flags+0x5f
tcp_input(c13fac00,14,c063dba8,c13fac00,3) at tcp_input+0x423
ip_input(c13fac00,c13fac00,0,c13fac00,4e) at ip_input+0x933
ether_demux(c1428000,c13fac00,800,c1428000,c1399800) at ether_demux+0x228
ether_input(c1428000,c13fac00,c067fa20,436,0) at ether_input+0x19a
ieee80211_input(c1428254,c13fac00,c1399800,2d,b79650e8) at 
ieee80211_input+0xd39
wi_intr(c1428000,0,0,c1223be8,0) at wi_intr+0x72d
ithread_loop(c121c700,cdf0bd48,28,c121c700,c04c4b76) at ithread_loop+0x113
fork_exit(c04c4b76,c121c700,cdf0bd48) at fork_exit+0x9b
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xcdf0bd7c, ebp = 0 ---
**********************************************************************

ddb (copied by hand):
**********************************************************************
KDB: enter: manual escape to debugger
[thread pid 13 tid 100004 ]
Stopped at      kdb_enter+0x2a: leave
db> show pcpu
cpuid        = 0
curthread    = 0xc121f8a0: pid 13 "irq 1: atkbd0"
curpcb       = 0xcdee4da0
fpcurthread  = none
idlethread   = 0xc121f5c0: pid 11 "idle"
APIC ID      = 0
currentldt   = 0x28
spin locks help:
db> where
Tracing pid 13 tid 100004 td 0xc121f8a0
kdb_enter(c0661268,c04f2c08,c06451bf,2,c0717a60) at kdb_enter+0x2a
scgetc(c06961c0,c137fa00,c0701980,c121f8a0,cdee4ce0) at scgetc+0x480
sckbdevent(c0701980,0,c0717a60) at sckbdevent_0x183
atkdb_intr(c0701980,0,cdee4d18,c04c4c89,c0701980) at atkdb_intr+0x1f
atkdb_isa_intr(c0701980,0,0,c121e7f0,0) at atkdb_isa_intr+0x16
ithread_loop(c121cb80,cdee4d48,28,c121cb80,c04c4b76) at ithread_loop+0x113
fork_exit(c04c4b76,c121cb80,cdee4d48) at fork_exit+0x9b
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xcdee4d7c, ebp = 0 ---
db> show alllocks
Process 520 (ssh) thread 0xc146f8a0 (100069)
exclusive sleep mutex inp (tcpinp) r = 0 (0xc14f1a68) locked _at_ 
/usr/src/sys/netinet/tcp_usrreq.c:603
Process 22 (irq11: cbb0 fxp0++) thread 0xc1220cf0 (100014)
exclusive sleep mutex tcp r = 0 (0xc06f512c) locked _at_ 
/usr/src/sys/netinet/tcp_input.c:614
exclusive sleep mutex wi0 (network driver) r = 0 (0xc1428d4c) locked _at_ 
/usr/src/sys/dev/wi/if_wi.c:601
Process 13 (irq1: atkbd0) thread 0xc121f8a0 (100004)
exclusive sleep mutex Giant r = 0 (0xc06a8080) locked _at_ 
/usr/src/sys/kern/kern_intr.c:545
db> call doadump
Dumping 319 MB
  16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304
Dump complete
0xf
db> reset
**********************************************************************

kgdb:
**********************************************************************
[GDB will not be able to debug user-mode threads: 
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
#0  doadump () at pcpu.h:164
164     pcpu.h: No such file or directory.
         in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:164
#1  0xc044061f in db_fncall (dummy1=-1068587398, dummy2=0, dummy3=-1,
     dummy4=0xcdee4b04 "") at /usr/src/sys/ddb/db_command.c:531
#2  0xc0440994 in db_command_loop () at /usr/src/sys/ddb/db_command.c:349
#3  0xc04421d2 in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:221
#4  0xc04ea9c8 in kdb_trap (type=3, code=0, tf=0xc06aa560)
     at /usr/src/sys/kern/subr_kdb.c:421
#5  0xc06154e1 in trap (frame=
       {tf_fs = -840040424, tf_es = -1068630000, tf_ds = -1067188208, 
tf_edi = -1066304928, tf_esi = -1066318208, tf_ebp = -840020876
, tf_isp = -840020896, tf_ebx = -1066304928, tf_edx = 1, tf_ecx = 
-1067148970, tf_eax = -1067052440, tf_trapno = 3, tf_err = 0, tf_e
ip = -1068587398, tf_cs = 8, tf_eflags = 524870, tf_esp = -840020828, 
tf_ss = -1067453223}) at /usr/src/sys/i386/i386/trap.c:573
#6  0xc0606e6a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xcdee0018 in ?? ()
#8  0xc04e0010 in umtxq_signal (key=0xc0714680, n_wake=0)
     at /usr/src/sys/kern/kern_umtx.c:155
#9  0xc05ff4d9 in scgetc (sc=0xc0717a60, flags=2)
     at /usr/src/sys/dev/syscons/syscons.c:3296
#10 0xc05ff764 in sckbdevent (thiskbd=0xc0701980, event=0, arg=0xc0717a60)
     at /usr/src/sys/dev/syscons/syscons.c:638
#11 0xc05f1ed4 in atkbd_intr (kbd=0xc0701980, arg=0x0)
     at /usr/src/sys/dev/kbd/atkbd.c:480
#12 0xc061bda7 in atkbd_isa_intr (arg=0x0) at 
/usr/src/sys/isa/atkbd_isa.c:177
---Type <return> to continue, or q <return> to quit---
#13 0xc04c4c89 in ithread_loop (arg=0xc121cb80)
     at /usr/src/sys/kern/kern_intr.c:546
#14 0xc04c419c in fork_exit (callout=0xc04c4b76 <ithread_loop>,
     arg=0xc121cb80, frame=0xcdee4d48) at /usr/src/sys/kern/kern_fork.c:790
#15 0xc0606ecc in fork_trampoline () at 
/usr/src/sys/i386/i386/exception.s:208
(kgdb) quit
**********************************************************************
Received on Sat Mar 26 2005 - 22:10:38 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:30 UTC