Re: FreeBSD 7, bridge, PF and syn flood = very bad performance

On Friday 25 January 2008, Stefan Lambrev wrote:
> Greetings,
>
> Does anyone try to see PF with "keep state" in action when under syn
> flood attack?
> I tried to get some help in freebsd-pf_at_, because the test firewall,
> that I build hardly can handle 2-5MB/s syn flood.
> Unfortunately I do not saw useful advice.
> The problem is that a quad core bridge firewall running freebsd 7 amd64
> with PF is near useless and can't handle "small" SYN ddos.
>
> Here is the schema that I'm testing:
> web server (freebsd) - freebsd (bridged interfaces) - gigabit switch -
> clients + flooders
> In this configuration ~25MB/s syn flood (and I think this limit is
> because of my switch) is not a problem and the web server responds
> without a problem.
> With this configuration netperf -l 610 -p 10303 -H 10.3.3.1 shows
> 116MB/s stable speed , so I guess there are no problems with cables,
> hardware and etc :)
>
> But when I start pf (see below the config file) the traffic drops to
> 2-3MB/s and the web server is hardly accessible.
> It seems that device polling helps a lot in this situation, and at
> least the bridge firewall is accessible. Without "polling" the firewall
> is so heavily loaded
> that even commands like "date" take few seconds to finish, with 2 cores
> at ~100% idle at same time.
>
> I have "flat profiles" from hwpmc, and I think it indicates a problem:
>
> (bridge, pf enabled, polling enabled, sched_ule - I have profiles and
> for other combinations too if needed)
>   %   cumulative   self              self     total
>  time   seconds   seconds    calls  ms/call  ms/call  name
>  24.0  268416.00 268416.00        0  100.00%           _mtx_lock_sleep

Can you build a kernel with LOCK_PROFILING and try to figure out which 
lock is causing this?

> [1] 6.7  343572.50 75156.50        0  100.00%
> pf_state_compare_ext_gwy [2]
>   6.7  418405.50 74833.00        0  100.00%           pf_src_compare
> [3] 3.9  462298.50 43893.00        0  100.00%
> pf_state_compare_lan_ext [4]
>   3.6  503019.50 40721.00        0  100.00%           pf_test [5]
>   3.2  538671.50 35652.00        0  100.00%           pf_test_tcp [6]
>   3.1  573743.50 35072.00        0  100.00%           bcopy [7]
>   2.3  599406.50 25663.00        0  100.00%           bridge_pfil [8]
>   2.0  622096.50 22690.00        0  100.00%
> pf_state_compare_id [9]
>   2.0  644331.50 22235.00        0  100.00%           pfsync_pack_state
> [10] 1.5  661563.50 17232.00        0  100.00%           uma_zalloc_arg
> [11] 1.3  676296.50 14733.00        0  100.00%           bzero [12] 1.3
>  690842.00 14545.50        0  100.00%
> bus_dmamap_load_mbuf_sg [13]
>   1.2  704521.50 13679.50        0  100.00%           uma_zfree_arg
> [14] 1.2  717554.50 13033.00        0  100.00%
> bridge_rtnode_lookup [15]
>   1.2  730575.50 13021.00        0  100.00%           _rw_rlock [16]
>   1.1  743076.50 12501.00        0  100.00%           rn_match [17]
>   1.0  754016.50 10940.00        0  100.00%           bridge_forward
> [18] 1.0  764766.00 10749.50        0  100.00%           em_encap [19]
> 0.9  775400.00 10634.00        0  100.00%           _rw_runlock [20]
> ........
>
> Also when running without polling top -S shows that external em0
> interface's taskq takes 100% CPU
> The situation is even worse when running with SCHED_BSD.
>
> Here is my test pf.conf
> quick
> #macros
> ext_if="em0"
> int_if="em1"
> br_if="bridge0"
>
> www="10.3.3.1"
>
> #sets
> set skip on lo0
> set skip on $int_if
> set skip on $br_if
> set limit states 20000000
> set limit src-nodes 19000000
> set optimization aggressive
>
> table <abusive_hosts> persist file "/etc/abusive_hosts"
>
> block log quick from <abusive_hosts> to any
> block log quick from any to <abusive_hosts>
>
> pass in on $ext_if proto tcp from any to $www port { 80, 443 } flags
> S/SA keep state \
> (source-track rule, max-src-conn-rate 150/10, max-src-states 500,
> max-src-nodes 4000000, overload <abusive_hosts> flush global)



-- 
/"\  Best regards,                      | mlaier_at_freebsd.org
\ /  Max Laier                          | ICQ #67774661
 X   http://pf4freebsd.love2party.net/  | mlaier_at_EFnet
/ \  ASCII Ribbon Campaign              | Against HTML Mail and News