Panic and LOR

Hi,

I've been getting a very reproducable panic in -CURRENT with moderate
network traffic (including same-machine traffic, and especially
with ssh), and a LOR on boot.

kiwi# uname -a
FreeBSD kiwi.petecurry.net 7.0-CURRENT FreeBSD 7.0-CURRENT #1: Sun
Sep  9 11:01:03 CDT 2007 root_at_:/usr/obj/usr/src/sys/KIWIKERN  amd64

The source is from about 10:00 this morning, and my kernel config
and dmesg output is attached. It's been happening for at least a
few weeks, about as long as I've been running -CURRENT and had this
machine.

The LOR message is:
Sep  9 11:16:00 kiwi kernel: lock order reversal: (sleepable after non-sleepable)
Sep  9 11:16:00 kiwi kernel: 1st 0xffffffff807781a8 PFil hook read/write mutex (PFil hook read/write mutex) _at_ /usr/src/sys/net/pfil.c:73
Sep  9 11:16:00 kiwi kernel: 2nd 0xffffffffb1e2f840 ipf filter load/unload mutex (ipf filter load/unload mutex) _at_ /usr/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/fil.c:2419
Sep  9 11:16:00 kiwi kernel: KDB: stack backtrace:
Sep  9 11:16:00 kiwi kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
Sep  9 11:16:00 kiwi kernel: witness_checkorder() at witness_checkorder+0x5f8
Sep  9 11:16:00 kiwi kernel: _sx_slock() at _sx_slock+0x51
Sep  9 11:16:00 kiwi kernel: fr_check() at fr_check+0x61
Sep  9 11:16:00 kiwi kernel: pfil_run_hooks() at pfil_run_hooks+0xc0
Sep  9 11:16:00 kiwi kernel: ip_output() at ip_output+0x37d
Sep  9 11:16:00 kiwi kernel: udp_send() at udp_send+0x350
Sep  9 11:16:00 kiwi kernel: sosend_dgram() at sosend_dgram+0x21a
Sep  9 11:16:00 kiwi kernel: kern_sendit() at kern_sendit+0x122
Sep  9 11:16:00 kiwi kernel: sendit() at sendit+0xdc
Sep  9 11:16:00 kiwi kernel: sendto() at sendto+0x4d
Sep  9 11:16:00 kiwi kernel: syscall() at syscall+0x1ce
Sep  9 11:16:00 kiwi kernel: Xfast_syscall() at Xfast_syscall+0xab
Sep  9 11:16:00 kiwi kernel: --- syscall (133, FreeBSD ELF64, sendto), rip = 0x800c57f6c, rsp = 0x7fffffffec18, rbp = 0x14705c8 ---

The panic and backtrace messages are (hand-copied, hopefully accurate):
panic: Trying sleep, but thread marked as sleeping prohibited
cpuid = 0
KDB: enter: panic
[thread pid 20 tid 100012 ]
Stopped at      kdb_enter+0x31: leave
db> bt
Tracing pid 20 tid 100012 td 0xffffff0001102000
kdb_enter() at kdb_enter+0x31
panic() at panic+0x173
sleepq_add() at sleepq_add_0x2c1
_sx_xlock_hard() at _sx_xlock_kard+0x19d
_sx_xlock() at _sx_xlock+0xb8
fr_check() at fr_check+0xc05
pfil_run_hooks() at pfil_run_hooks+0xc0
ip_output() at ip_output+037d
tcp_output() at tcp_output+0xad6
tcp_timer_delack+0xcc
softclock() at softclock+0x297
ithread_loop() at ithread_loop+0xe0
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_tramoline+0xe
--- trap 0, rip= 0, rsp = 0xffffffffabeb1d30, rbp = 0 ---
db> 

I've got a crash dump, but kgdb reports:
kiwi# cd /usr/obj/usr/src/sys/KIWIKERN/
kiwi# kgdb kernel.debug /var/crash/vmcore.0
kgdb: kvm_read: invalid address (0x6461657268742074)
kgdb: kvm_read: invalid address (0x7320676e69797254)
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd".
Cannot access memory at address 0x1464
(kgdb) bt
#0  0x0000000000000000 in ?? ()
Cannot access memory at address 0x0
(kgdb) 

Please let me know what other information that I can provide or
anything that I can do to help with this issue.

Thanks,
- Pete Curry