LOR's, and a panic (ipf NAT related?)

From: Eric Anderson <anderson_at_freebsd.org>
Date: Sat, 21 Jul 2007 20:36:31 -0500
Today, on a -CURRENT from a few days ago (running ULE 3.0), I got a panic:

panic: Trying sleep, but thread marked as sleeping prohibited
cpuid = 0

#8  0xc074a474 in panic (fmt=0xc0a979a9 "Trying sleep, but thread marked 
as sleeping prohibited")
     at /usr/src/sys/kern/kern_shutdown.c:547
#9  0xc077abd2 in sleepq_add (wchan=0xc0e20980, lock=0x0, 
wmesg=0xc0e1b709 "ipf IP NAT rwlock", flags=3, queue=0)
     at /usr/src/sys/kern/subr_sleepqueue.c:289
#10 0xc07519ee in _sx_xlock_hard (sx=0xc0e20980, tid=3306307584, opts=0,
     file=0xc0e1b65e 
"/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c", 
line=4384)
     at /usr/src/sys/kern/kern_sx.c:548
#11 0xc0751d18 in _sx_xlock (sx=0xc0e20980, opts=0,
     file=0xc0e1b65e 
"/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c", 
line=4384) at sx.h:153


I also see these (maybe related) LOR's on bootup:

lock order reversal: (sleepable after non-sleepable)
  1st 0xc0e207f0 ipf nat io mutex (ipf nat io mutex) _at_ 
/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c:710
  2nd 0xc0e20980 ipf IP NAT rwlock (ipf IP NAT rwlock) _at_ 
/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_nat.c:1062
KDB: stack backtrace:
db_trace_self_wrapper(c0a96ace,e7a6ddc8,c0782f8e,c0a98fa0,c0e20980,...) 
at db_trace_self_wrapper+0x26
kdb_backtrace(c0a98fa0,c0e20980,c0e1b709,c0e1b709,c0e1b65e,...) at 
kdb_backtrace+0x29
witness_checkorder(c0e20980,9,c0e1b65e,426,4,...) at 
witness_checkorder+0x6de
_sx_xlock(c0e20980,0,c0e1b65e,426,0,...) at _sx_xlock+0x7d
fr_nat_ioctl(c55d26c0,8034723c,3,0,c561d880,...) at fr_nat_ioctl+0x6ab
fr_ioctlswitch(1,c55d26c0,8034723c,3,0,...) at fr_ioctlswitch+0x79
iplioctl(c53b3a00,8034723c,c55d26c0,3,c561d880,...) at iplioctl+0xd8
devfs_ioctl_f(c5643af8,8034723c,c55d26c0,c5740a00,c561d880,...) at 
devfs_ioctl_f+0xc9
kern_ioctl(c561d880,3,8034723c,c55d26c0,0,...) at kern_ioctl+0x243
ioctl(c561d880,e7a6ecfc,c,c,c0b40130,...) at ioctl+0x134
syscall(e7a6ed38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x28167e53, esp = 
0xbfbfec9c, ebp = 0xbfbfed08 ---


lock order reversal: (sleepable after non-sleepable)
  1st 0xc0bfb798 PFil hook read/write mutex (PFil hook read/write mutex) 
_at_ /usr/src/sys/net/pfil.c:73
  2nd 0xc0e20880 ipf filter load/unload mutex (ipf filter load/unload 
mutex) _at_ 
/usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/fil.c:2419
KDB: stack backtrace:
db_trace_self_wrapper(c0a96ace,e3cc299c,c0782f8e,c0a98fa0,c0e20880,...) 
at db_trace_self_wrapper+0x26
kdb_backtrace(c0a98fa0,c0e20880,c0e1b400,c0e1b400,c0e1c7c1,...) at 
kdb_backtrace+0x29
witness_checkorder(c0e20880,1,c0e1c7c1,973,c53f1000,...) at 
witness_checkorder+0x6de
_sx_slock(c0e20880,0,c0e1c7c1,973,c5126880,...) at _sx_slock+0x7d
fr_check(c57be024,14,c5269400,0,e3cc2ad0,...) at fr_check+0x5a
fr_check_wrapper(0,e3cc2ad0,c5269400,1,0,...) at fr_check_wrapper+0x3f
pfil_run_hooks(c0bfb780,e3cc2b24,c5269400,1,0,...) at pfil_run_hooks+0x88
ip_input(c5782200,c0652b32,800,c5269400,800,...) at ip_input+0x24d
netisr_dispatch(2,c5782200,46a266ac,89ca3,c5269400,...) at 
netisr_dispatch+0x73
ether_demux(c5269400,c5782200,3,0,3,...) at ether_demux+0x1f1
ether_input(c5269400,c5782200,18,c0780d4e,c5782200,...) at ether_input+0x37f
ieee80211_deliver_data(c526a22c,c57df000,c5782200,18,c0aa34ce,...) at 
ieee80211_deliver_data+0x13e
ieee80211_input(c526a22c,c5782200,c57df000,21,ffffffaa,...) at 
ieee80211_input+0x1159
ath_rx_proc(c526a000,1,c0a97ce7,52,c527bb9c,...) at ath_rx_proc+0x52d
taskqueue_run(c527bb80,c527bb9c,0,c0a8b649,0,...) at taskqueue_run+0x10b
taskqueue_thread_loop(c526b65c,e3cc2d38,c0a9069e,315,c5223000,...) at 
taskqueue_thread_loop+0x68
fork_exit(c077c220,c526b65c,e3cc2d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe3cc2d70, ebp = 0 ---


I have a vmcore if anyone wants me to debug further..

Eric
Received on Sat Jul 21 2007 - 23:36:38 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:15 UTC