Re: [CFT] SIFTR - Statistical Information For TCP Research: Uncle Lawrence needs YOU!

On 06/21/10 00:12, Fabian Keil wrote:
> Fabian Keil<freebsd-listen_at_fabiankeil.de>  wrote:
>
>> Lawrence Stewart<lstewart_at_freebsd.org>  wrote:
>>
>>> On 06/20/10 22:28, Fabian Keil wrote:
>
>>>> Taking pf (and altq) out of the picture doesn't seem to make
>>>> a difference.
>>>
>>> Wouldn't have expected it to. Will be very curious to know if the panic
>>> is triggered in GENERIC.
>>
>> It's not. I, too, get pfil.c related LORs though:
>>
>> lock order reversal:
>>   1st 0xffffffff80e5c568 PFil hook read/write mutex (PFil hook read/write mutex) _at_ /usr/src/sys/net/pfil.c:77
>>   2nd 0xffffffff80e5dd68 udp (udp) _at_ /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:3035
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
>> _witness_debugger() at _witness_debugger+0x2e
>> witness_checkorder() at witness_checkorder+0x81e
>> _rw_rlock() at _rw_rlock+0x5f
>> pf_socket_lookup() at pf_socket_lookup+0x1c5
>> pf_test_udp() at pf_test_udp+0x8b0
>> pf_test() at pf_test+0x1089
>> pf_check_in() at pf_check_in+0x39
>> pfil_run_hooks() at pfil_run_hooks+0xcf
>> ip_input() at ip_input+0x2ae
>> swi_net() at swi_net+0x151
>> intr_event_execute_handlers() at intr_event_execute_handlers+0x66
>> ithread_loop() at ithread_loop+0xb2
>> fork_exit() at fork_exit+0x12a
>> fork_trampoline() at fork_trampoline+0xe
>> --- trap 0, rip = 0, rsp = 0xffffff8000044d30, rbp = 0 ---
>> lock order reversal:
>>   1st 0xffffffff80e5c568 PFil hook read/write mutex (PFil hook read/write mutex) _at_ /usr/src/sys/net/pfil.c:77
>>   2nd 0xffffffff80e5d788 tcp (tcp) _at_ /usr/src/sys/modules/siftr/../../netinet/siftr.c:698
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
>> _witness_debugger() at _witness_debugger+0x2e
>> witness_checkorder() at witness_checkorder+0x81e
>> _rw_rlock() at _rw_rlock+0x5f
>> siftr_chkpkt() at siftr_chkpkt+0x3c4
>> pfil_run_hooks() at pfil_run_hooks+0xcf
>> ip_input() at ip_input+0x2ae
>> swi_net() at swi_net+0x151
>> intr_event_execute_handlers() at intr_event_execute_handlers+0x66
>> ithread_loop() at ithread_loop+0xb2
>> fork_exit() at fork_exit+0x12a
>> fork_trampoline() at fork_trampoline+0xe
>> --- trap 0, rip = 0, rsp = 0xffffff8000044d30, rbp = 0 ---
>>
>> My custom kernel normally doesn't have INVARIANTS and WITNESS
>> enabled, so I'll try to enable them next.
>
> The culprit seem to be non-default KTR settings in the kernel
> while loading alq as a module. With the following change siftr
> works with my non-GENERIC kernel, too:
>
> commit f43b8b5171c858df7b419f6a695e9e3b53531a8e
> Author: Fabian Keil<fk_at_fabiankeil.de>
> Date:   Sun Jun 20 15:43:01 2010 +0200
>
>      Disable KTR changes.
>
> diff --git a/sys/amd64/conf/ZOEY b/sys/amd64/conf/ZOEY
> index 6fb3480..c584317 100644
> --- a/sys/amd64/conf/ZOEY
> +++ b/sys/amd64/conf/ZOEY
> _at__at_ -16,11 +16,11 _at__at_ options     ATA_CAM
>   device          atapicam
>   options         SC_KERNEL_CONS_ATTR=(FG_GREEN|BG_BLACK)
>
> -options         KTR
> -options         KTR_ENTRIES=262144
> -options         KTR_COMPILE=(KTR_SCHED)
> -options         KTR_MASK=(KTR_SCHED)
> -options         KTR_CPUMASK=0x3
> +#options         KTR
> +#options         KTR_ENTRIES=262144
> +#options         KTR_COMPILE=(KTR_SCHED)
> +#options         KTR_MASK=(KTR_SCHED)
> +#options         KTR_CPUMASK=0x3
>
>   options ACCEPT_FILTER_HTTP
>   makeoptions WITH_CTF=yes

This smells very fishy. Without "options KTR_ALQ", KTR shouldn't even 
care if ALQ exists or not. Not only that, but ALQ isn't even used in 
siftr_chkpkt and you clearly manage to successfully use ALQ to write the 
module load message to the log file. Hmmmm...

Thanks for taking the time to find the culprit though - I'll see if I 
can reproduce here. Could you try another thing for me and see if 
reducing "options KTR_ENTRIES=262144" down to a smaller number (maybe 
4096?) and leaving all the other KTR options as they are above (but 
uncommented) makes any difference? The ktr(4) man page indicates the 
default is 8192 entries and I'm curious if the your allocation of so 
many additional entries is making something unhappy.

Thanks again for your time helping with this, I really appreciate it.

Cheers,
Lawrence