Re: Panic and LOR

This is a known issue. sx locks aren't for use in the I/O path. The
firewall maintainer chooses to use them. This might be avoidable in
the future if we can establish a mechanism for preventing pfil from
being unloaded without protecting it with an rwlock.

 -Kip



On 9/9/07, Pete Curry <mail_at_petecurry.net> wrote:
> Hi,
>
> I've been getting a very reproducable panic in -CURRENT with moderate
> network traffic (including same-machine traffic, and especially
> with ssh), and a LOR on boot.
>
> kiwi# uname -a
> FreeBSD kiwi.petecurry.net 7.0-CURRENT FreeBSD 7.0-CURRENT #1: Sun
> Sep  9 11:01:03 CDT 2007 root_at_:/usr/obj/usr/src/sys/KIWIKERN  amd64
>
> The source is from about 10:00 this morning, and my kernel config
> and dmesg output is attached. It's been happening for at least a
> few weeks, about as long as I've been running -CURRENT and had this
> machine.
>
> The LOR message is:
> Sep  9 11:16:00 kiwi kernel: lock order reversal: (sleepable after non-sleepable)
> Sep  9 11:16:00 kiwi kernel: 1st 0xffffffff807781a8 PFil hook read/write mutex (PFil hook read/write mutex) _at_ /usr/src/sys/net/pfil.c:73
> Sep  9 11:16:00 kiwi kernel: 2nd 0xffffffffb1e2f840 ipf filter load/unload mutex (ipf filter load/unload mutex) _at_ /usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/fil.c:2419
> Sep  9 11:16:00 kiwi kernel: KDB: stack backtrace:
> Sep  9 11:16:00 kiwi kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> Sep  9 11:16:00 kiwi kernel: witness_checkorder() at witness_checkorder+0x5f8
> Sep  9 11:16:00 kiwi kernel: _sx_slock() at _sx_slock+0x51
> Sep  9 11:16:00 kiwi kernel: fr_check() at fr_check+0x61
> Sep  9 11:16:00 kiwi kernel: pfil_run_hooks() at pfil_run_hooks+0xc0
> Sep  9 11:16:00 kiwi kernel: ip_output() at ip_output+0x37d
> Sep  9 11:16:00 kiwi kernel: udp_send() at udp_send+0x350
> Sep  9 11:16:00 kiwi kernel: sosend_dgram() at sosend_dgram+0x21a
> Sep  9 11:16:00 kiwi kernel: kern_sendit() at kern_sendit+0x122
> Sep  9 11:16:00 kiwi kernel: sendit() at sendit+0xdc
> Sep  9 11:16:00 kiwi kernel: sendto() at sendto+0x4d
> Sep  9 11:16:00 kiwi kernel: syscall() at syscall+0x1ce
> Sep  9 11:16:00 kiwi kernel: Xfast_syscall() at Xfast_syscall+0xab
> Sep  9 11:16:00 kiwi kernel: --- syscall (133, FreeBSD ELF64, sendto), rip = 0x800c57f6c, rsp = 0x7fffffffec18, rbp = 0x14705c8 ---
>
> The panic and backtrace messages are (hand-copied, hopefully accurate):
> panic: Trying sleep, but thread marked as sleeping prohibited
> cpuid = 0
> KDB: enter: panic
> [thread pid 20 tid 100012 ]
> Stopped at      kdb_enter+0x31: leave
> db> bt
> Tracing pid 20 tid 100012 td 0xffffff0001102000
> kdb_enter() at kdb_enter+0x31
> panic() at panic+0x173
> sleepq_add() at sleepq_add_0x2c1
> _sx_xlock_hard() at _sx_xlock_kard+0x19d
> _sx_xlock() at _sx_xlock+0xb8
> fr_check() at fr_check+0xc05
> pfil_run_hooks() at pfil_run_hooks+0xc0
> ip_output() at ip_output+037d
> tcp_output() at tcp_output+0xad6
> tcp_timer_delack+0xcc
> softclock() at softclock+0x297
> ithread_loop() at ithread_loop+0xe0
> fork_exit() at fork_exit+0x12a
> fork_trampoline() at fork_tramoline+0xe
> --- trap 0, rip= 0, rsp = 0xffffffffabeb1d30, rbp = 0 ---
> db>
>
> I've got a crash dump, but kgdb reports:
> kiwi# cd /usr/obj/usr/src/sys/KIWIKERN/
> kiwi# kgdb kernel.debug /var/crash/vmcore.0
> kgdb: kvm_read: invalid address (0x6461657268742074)
> kgdb: kvm_read: invalid address (0x7320676e69797254)
> [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd".
> Cannot access memory at address 0x1464
> (kgdb) bt
> #0  0x0000000000000000 in ?? ()
> Cannot access memory at address 0x0
> (kgdb)
>
> Please let me know what other information that I can provide or
> anything that I can do to help with this issue.
>
> Thanks,
> - Pete Curry
>
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
>
>