Re: ipfw: LOR/panic with uid rules

On Thursday 25 September 2008 22:51:00 Robert Watson wrote:
> On Thu, 25 Sep 2008, Stefan Ehmann wrote:
> > Hmm, just obtained a new dump which was the same. Did a normal "make
> > kernel", so source/kernel should be in sync
> >
> > This is the version:
> >
> > __FBSDID("$FreeBSD: src/sys/netinet/tcp_input.c,v 1.382 2008/09/24
> > 11:07:03 rwatson Exp $");
> >
> > What doesn't match? I only checked this and it looks okay to me
>
> Indeed, it looks like I had my own source synchronization issue :-).
>
> This backtrace is differen from the previous one, and is for a different
> instance of the same bug.  I believe I've corrected it with this change:
>
> rwatson     2008-09-25 17:26:54 UTC

Thanks, that definitely is an improvement for me.

My initial test case doesn't cause a panic any longer. The LORs remain.

Under a bit heavier load, I get a new LOR and panic after some minutes.

lock order reversal:
 1st 0xc4c9ee94 tcp_sc_head (tcp_sc_head) _at_ /usr/src/sys/kern/kern_mutex.c:137
 2nd 0xc0e59fd8 PFil hook read/write mutex (PFil hook read/write mutex) _at_ 
/usr/src/sys/net/pfil.c:74
KDB: stack backtrace:
db_trace_self_wrapper(c0bad7c2,c45aca48,c082cf95,4,c0ba916b,...) at 
db_trace_self_wrapper+0x26
kdb_backtrace(4,c0ba916b,c0bb97db,c4879d08,c45acaa4,...) at kdb_backtrace+0x29
_witness_debugger(c0bb0077,c0e59fd8,c0bb97f3,c4879d08,c0bb97db,...) at 
_witness_debugger+0x25
witness_checkorder(c0e59fd8,1,c0bb97db,4a,0,...) at witness_checkorder+0x810
_rm_rlock_debug(c0e59fd8,c45acaec,c0bb97db,4a,c089e366,...) at 
_rm_rlock_debug+0x38
pfil_run_hooks(c0e59fc0,c45acb78,c4b0a000,2,0,...) at pfil_run_hooks+0x3f
ip_output(c4cbba00,0,0,0,0,...) at ip_output+0x872
syncache_respond(c5376b00,0,0,0,c45acc48,...) at syncache_respond+0x3a9
syncache_timer(c4c9ee94,1,c0bab9c2,16b,c0cf3034,...) at syncache_timer+0x147
softclock(c0cf3000,c45accc8,c07e0ec4,c0cf69c0,c4905938,...) at softclock+0x24a
intr_event_execute_handlers(c48c07d4,c4905900,c0ba6d4b,4dd,c4905970,...) at 
intr_event_execute_handlers+0x125
ithread_loop(c48bf4d0,c45acd38,c0ba6abd,322,c48c07d4,...) at ithread_loop+0x9f
fork_exit(c07d0920,c48bf4d0,c45acd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc45acd70, ebp = 0 ---

(kgdb) bt
#0  doadump () at pcpu.h:221
#1  0xc04bb929 in db_fncall (dummy1=1, dummy2=0, dummy3=0, 
    dummy4=0xc4732338 "") at /usr/src/sys/ddb/db_command.c:549
#2  0xc04bbf31 in db_command (last_cmdp=0xc0cc06dc, cmd_table=0x0, dopager=1)
    at /usr/src/sys/ddb/db_command.c:446
#3  0xc04bc08a in db_command_loop () at /usr/src/sys/ddb/db_command.c:499
#4  0xc04bda3d in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:228
#5  0xc081b7b6 in kdb_trap (type=3, code=0, tf=0xc47324e0)
    at /usr/src/sys/kern/subr_kdb.c:534
#6  0xc0aff466 in trap (frame=0xc47324e0) at /usr/src/sys/i386/i386/trap.c:694
#7  0xc0ae3adb in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#8  0xc081b93a in kdb_enter (why=0xc0baa61b "panic", msg=0xc0baa61b "panic")
    at cpufunc.h:70
#9  0xc07ee7fc in panic (fmt=0xc0baa121 "%s (%s): wlock already held _at_ %s:%d")
    at /usr/src/sys/kern/kern_shutdown.c:556
#10 0xc07eccd6 in _rw_rlock (rw=0xc0e5acec, 
    file=0xc103ceed "/usr/src/sys/modules/ipfw/../../netinet/ip_fw2.c", 
    line=2020) at /usr/src/sys/kern/kern_rwlock.c:283
#11 0xc103b92a in ipfw_chk (args=0xc47328a8)
    at /usr/src/sys/modules/ipfw/../../netinet/ip_fw2.c:2020
#12 0xc103c4c8 in ipfw_check_out (arg=0x0, m0=0xc47329cc, ifp=0xc4b0a000, 
    dir=2, inp=0xc50fe420)
    at /usr/src/sys/modules/ipfw/../../netinet/ip_fw_pfil.c:253
#13 0xc08992a8 in pfil_run_hooks (ph=0xc0e59fc0, mp=0xc4732a3c, 
    ifp=0xc4b0a000, dir=2, inp=0xc50fe420) at /usr/src/sys/net/pfil.c:79
#14 0xc08e1602 in ip_output (m=0xc4cbe100, opt=0x0, ro=0xc4732a44, flags=0, 
    imo=0x0, inp=0xc50fe420) at /usr/src/sys/netinet/ip_output.c:452
#15 0xc094842e in tcp_twrespond (tw=0xc53d9104, flags=Variable "flags" is not 
available.
)
    at /usr/src/sys/netinet/tcp_timewait.c:602
#16 0xc0948886 in tcp_twcheck (inp=0xc50fe420, to=0xc4732b5c, th=0xc4e2382a, 
    m=0xc4e05700, tlen=0) at /usr/src/sys/netinet/tcp_timewait.c:407
#17 0xc093d66a in tcp_input (m=0xc4e05700, off0=20)
    at /usr/src/sys/netinet/tcp_input.c:554
#18 0xc08dfc10 in ip_input (m=0xc4e05700)
    at /usr/src/sys/netinet/ip_input.c:666
#19 0xc0898aa3 in netisr_dispatch (num=2, m=0xc4e05700)
    at /usr/src/sys/net/netisr.c:178
#20 0xc08929f1 in ether_demux (ifp=0xc4b0a000, m=0xc4e05700)
    at /usr/src/sys/net/if_ethersubr.c:842
#21 0xc0892e5f in ether_input (ifp=0xc4b0a000, m=0xc4e05700)
    at /usr/src/sys/net/if_ethersubr.c:700
#22 0xc076503e in vr_intr (arg=0xc4b22000) at /usr/src/sys/dev/vr/if_vr.c:1414
#23 0xc07cfd45 in intr_event_execute_handlers (p=0xc48c07d4, ie=0xc4905a80)
    at /usr/src/sys/kern/kern_intr.c:1134
#24 0xc07d09bf in ithread_loop (arg=0xc4b29a10)
    at /usr/src/sys/kern/kern_intr.c:1147
#25 0xc07cdb08 in fork_exit (callout=0xc07d0920 <ithread_loop>, 
    arg=0xc4b29a10, frame=0xc4732d38) at /usr/src/sys/kern/kern_fork.c:810
#26 0xc0ae3b50 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:270