deadlocks in recent 6-CURRENT

Since switching to PF (few weeks ago) I'm frequently experiencing deadlocks.
breaking to DDB works, and ps shows lock in PF, like here:

43273 c1ee57e0 1001 42753 43273 0004002 [LOCK pf task mtx c1c45040] dia

Setting debug.mpsafenet to 0 helps a little (system runs without
deadlock for few days) but doesn't eliminate it.

PREEMPTION is on

WITNESS show two kinds of LOR's:

 1st 0xc1888414 inp (tcpinp) _at_ /usr/src/sys/netinet/tcp_syncache.c:1086
 2nd 0xc06b9400 pf task mtx (pf task mtx) _at_ /usr/src/sys/contrib/pf/net/pf.c:5891
KDB: stack backtrace:
witness_checkorder(c06b9400,9,c0668a25,1703,7) at witness_checkorder+0x388
_mtx_lock_flags(c06b9400,0,c0668a25,1703,cbc9c8f8) at _mtx_lock_flags+0x37
pf_test(2,c15ce000,cbc9ca08,c1888384,c163cba0) at pf_test+0x72
pf_check_out(0,cbc9ca08,c15ce000,2,c1888384) at pf_check_out+0x4e
pfil_run_hooks(c06eafe0,cbc9ca94,c15ce000,2,c1888384) at pfil_run_hooks+0x145
ip_output(c16bad00,0,cbc9ca60,0,0) at ip_output+0x299
syncache_respond(cbc9cb20,c06eba40,1c,8,78) at syncache_respond+0x26a
syncache_add(cbc9cbac,cbc9cbfc,c17a8034,cbc9cba8,c1773b00) at syncache_add+0x428
tcp_input(c1773b00,14,c15ce000,1,0) at tcp_input+0x1a87
ip_input(c1773b00,0,c06797c5,e8,18) at ip_input+0xd5
netisr_processqueue(cbc9ccdc,80246,c06bf1a0,8,c14f3c40) at netisr_processqueue+0x15
swi_net(0,0,c0670763,257,0) at swi_net+0x9f
ithread_loop(c14fa500,cbc9cd48,c06705b6,318,c14fa500) at ithread_loop+0x1d2
fork_exit(c04d53e0,c14fa500,cbc9cd48) at fork_exit+0x70
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xcbc9cd7c, ebp = 0 ---

 1st 0xc06eb44c tcp (tcp) _at_ /usr/src/sys/netinet/tcp_input.c:615
 2nd 0xc06b9400 pf task mtx (pf task mtx) _at_ /usr/src/sys/contrib/pf/net/pf.c:5891
KDB: stack backtrace:
witness_checkorder(c06b9400,9,c0668a25,1703,0) at witness_checkorder+0x388
_mtx_lock_flags(c06b9400,0,c0668a25,1703,c04e1dc9) at _mtx_lock_flags+0x37
pf_test(2,c16cf014,cbc9ca7c,0,c163cba0) at pf_test+0x72
pf_check_out(0,cbc9ca7c,c16cf014,2,0) at pf_check_out+0x4e
pfil_run_hooks(c06eafe0,cbc9cb08,c16cf014,2,0) at pfil_run_hooks+0x145
ip_output(c16b6a00,0,cbc9cad4,0,0,0,0,0,c06a495c,c8) at ip_output+0x299
tcp_respond(0,c16b6a34,c16b6a48,c16b6a00,e5623f88) at tcp_respond+0x16e
tcp_input(c16b6a00,14,c16cf014,1,0) at tcp_input+0x91e
ip_input(c16b6a00,0,c06797c5,e8,18) at ip_input+0xd5
netisr_processqueue(cbc9ccdc,80246,c06bf1a0,8,c14f3c40) at netisr_processqueue+0x15
swi_net(0,0,c0670763,257,0) at swi_net+0x9f
ithread_loop(c14fa500,cbc9cd48,c06705b6,318,c14fa500) at ithread_loop+0x1d2
fork_exit(c04d53e0,c14fa500,cbc9cd48) at fork_exit+0x70
fork_trampoline() at fork_trampoline+0x8

I can provide core dump along with kernel.debug and configuration files.



Michal