Re: FreeBSD 7, bridge, PF and syn flood = very bad performance

From: arc_gabriel <m.pagulayan_at_auckland.ac.nz>
Date: Mon, 16 Jun 2008 14:44:09 -0700 (PDT)
Hi Guys,

I am in the same boat as you guys are. 

I am using pf from 7.0-RELEASE FreeBSD 7.0-RELEASE 

Hardware Specs are 
IBM x3655
Interfaces: em0 and em1 (intel 1G cards)
FW Setup: As Bridge

I get very high load on CPU whenever em1 taskq and em0 taskq reaches it
peaks. I noticed this when traffic goes to about more than 17MB/s or I can
see drops as well on em1(external interface) when packets go over 14K
packets/s. Which is a bit slow for this new hardware. 

But the other thing we setup with PF is Altq. Basically We have queues on
both the em0(internal) and em1(externaL) interfaces. May this is another
bottleneck? 

I was just wondering on how you remedied this problem.  I am badly in need
of solutions. Our internet has been crappy for the past week because when
the CPU load goes high it drops heaps of packets. 

Best Regads, 

Mark

Max Laier wrote:
> 
> On Sunday 27 January 2008, Stefan Lambrev wrote:
>> Greetings,
>>
>> Max Laier wrote:
>>
>> -cut-
>>
>> >> Well I think the interesting lines from this experiment are:
>> >> max              total            wait_total       count   avg
>> >> wait_avg     cnt_hold     cnt_lock name
>> >>     39            25328476     70950955     9015860     2     7
>> >> 5854948      6309848 /usr/src/sys/contrib/pf/net/pf.c:6729 (sleep
>> >> mutex:pf task mtx)
>> >> 936935        10645209          350          50 212904     7
>> >> 110           47 /usr/src/sys/contrib/pf/net/pf.c:980 (sleep
>> >> mutex:pf task mtx)
>> >
>> > Yeah, those two mostly are the culprit, but a quick fix is not really
>> > available.  You can try to "set timeout interval" to something bigger
>> > (e.g. 60 seconds) which will decrease the average hold time of the
>> > second lock instance at the cost of increased peak memory usage.
>>
>> I'll try and this. At least memory doesn't seems to be a problem :)
>>
>> > I have the ideas how to fix this, but it will take much much more
>> > time than I currently have for FreeBSD :-\  In general this requires
>> > a bottom up redesign of pf locking and some data structures involved
>> > in the state tree handling.
>> >
>> > The first(=main) lock instance is also far from optimal (i.e. pf is a
>> > congestion point in the bridge forwarding path).  For this I have
>> > also a plan how to make at least state table lookups run in parallel
>> > to some extend, but again the lack of free time to spend coding
>> > prevents me from doing it at the moment :-\
>>
>> Well, now we know where the issue is. The same problem seems to affect
>> synproxy state btw.
>> Can I expect better performance with IPFW's dynamic rules?
> 
> Not significantly better, I'd predict.  IPFW's dynamic rules are also 
> protected by a single mutex leading to similar congestion problems as pf.  
> There should be a measureable constant improvement as IPFW does much less 
> sanity checks.  i.e. better performance at the expense of less security.  
> It really depends on your needs which is better suited for your setup.
> 
>> I wonder how one can protect himself on gigabit network and service
>> more then 500pps.
>> For example in my test lab I see incoming ~400k packets per second, but
>> if I activate PF,
>> I see only 130-140k packets per second. Is this expected behavior, if
>> PF cannot handle so many packets?
> 
> As you can see from the hwpmc trace starting this thread, we don't spend 
> that much time in pf.  The culprit is the pf task mutext, which forces 
> serialization in pf congesting the whole forward path.  Under different 
> circumstances pf can handle more pps.
> 
>> The missing 250k+ are not listed as discarded or other errors, which is
>> weird.
> 
> As you slow down the forwarding protocols like TCP will automatically slow 
> down. Unless you have UDP bombs blasting at your network this is quite 
> usual behavior.
> 
> -- 
> /"\  Best regards,                      | mlaier_at_freebsd.org
> \ /  Max Laier                          | ICQ #67774661
>  X   http://pf4freebsd.love2party.net/  | mlaier_at_EFnet
> / \  ASCII Ribbon Campaign              | Against HTML Mail and News
> 
>  
> 

-- 
View this message in context: http://www.nabble.com/FreeBSD-7%2C-bridge%2C-PF-and-syn-flood-%3D-very-bad-performance-tp15093449p17873869.html
Sent from the freebsd-current mailing list archive at Nabble.com.
Received on Mon Jun 16 2008 - 20:04:26 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:32 UTC