Goo Day, Max Laier wrote: > On Saturday 26 January 2008, Stefan Lambrev wrote: > >> Max Laier wrote: >> >>> On Friday 25 January 2008, Stefan Lambrev wrote: >>> >>>> Greetings, >>>> >>>> Does anyone try to see PF with "keep state" in action when under syn >>>> flood attack? >>>> I tried to get some help in freebsd-pf_at_, because the test firewall, >>>> that I build hardly can handle 2-5MB/s syn flood. >>>> Unfortunately I do not saw useful advice. >>>> The problem is that a quad core bridge firewall running freebsd 7 >>>> amd64 with PF is near useless and can't handle "small" SYN ddos. >>>> >>>> Here is the schema that I'm testing: >>>> web server (freebsd) - freebsd (bridged interfaces) - gigabit switch >>>> - clients + flooders >>>> In this configuration ~25MB/s syn flood (and I think this limit is >>>> because of my switch) is not a problem and the web server responds >>>> without a problem. >>>> With this configuration netperf -l 610 -p 10303 -H 10.3.3.1 shows >>>> 116MB/s stable speed , so I guess there are no problems with cables, >>>> hardware and etc :) >>>> >>>> But when I start pf (see below the config file) the traffic drops to >>>> 2-3MB/s and the web server is hardly accessible. >>>> It seems that device polling helps a lot in this situation, and at >>>> least the bridge firewall is accessible. Without "polling" the >>>> firewall is so heavily loaded >>>> that even commands like "date" take few seconds to finish, with 2 >>>> cores at ~100% idle at same time. >>>> >>>> I have "flat profiles" from hwpmc, and I think it indicates a >>>> problem: >>>> >>>> (bridge, pf enabled, polling enabled, sched_ule - I have profiles >>>> and for other combinations too if needed) >>>> % cumulative self self total >>>> time seconds seconds calls ms/call ms/call name >>>> 24.0 268416.00 268416.00 0 100.00% >>>> _mtx_lock_sleep >>>> >>> Can you build a kernel with LOCK_PROFILING and try to figure out >>> which lock is causing this? >>> >> Yes I can build kernel with LOCK_PROFILING. >> But I have no idea how to use it :) >> Can you point me to some documentation? >> > > man LOCK_PROFILING > > basically: > # sysctl debug.lock.prof.enable=1 && sleep 60 && \ > sysctl debug.lock.prof.enable=0 && \ > sysctl debug.lock.prof.stats > log > > while under attack to sample one minute of lock statistics. > > Well I think the interesting lines from this experiment are: max total wait_total count avg wait_avg cnt_hold cnt_lock name 39 25328476 70950955 9015860 2 7 5854948 6309848 /usr/src/sys/contrib/pf/net/pf.c:6729 (sleep mutex:pf task mtx) 936935 10645209 350 50 212904 7 110 47 /usr/src/sys/contrib/pf/net/pf.c:980 (sleep mutex:pf task mtx) 41 10528492 1422891 1492295 7 0 155627 216812 /usr/src/sys/dev/em/if_em.c:980 (sleep mutex:em1) 26 5894103 2275517 2254004 2 1 427066 715901 /usr/src/sys/net/if_bridge.c:2082 (sleep mutex:if_bridge) 34 5466679 118638 761766 7 0 1198 5794 /usr/src/sys/dev/em/if_em.c:980 (sleep mutex:em0) 24 4274965 1952823 2253930 1 0 201352 691434 /usr/src/sys/net/if_bridge.c:1991 (sleep mutex:if_bridge) 28 3067953 800284 1492265 2 0 113423 294092 /usr/src/sys/net/if_bridge.c:1674 (sleep mutex:em1) 776401 1972047 0 69 28580 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:145 (sx:so_snd_sx) 775844 1970701 0 69 28560 0 1 0 /usr/src/sys/netinet/tcp_usrreq.c:779 (sleep mutex:inp) 22 1552808 922 761744 2 0 6 405 /usr/src/sys/dev/em/if_em.c:949 (sleep mutex:em0) 19 1508717 94 761736 1 0 51 24 /usr/src/sys/net/if_bridge.c:1674 (sleep mutex:em0) 15 713930 7045 590468 1 0 1778 3364 /usr/src/sys/kern/kern_timeout.c:419 (spin mutex:callout) 9 693209 4395 579397 1 0 1305 2129 /usr/src/sys/kern/kern_timeout.c:500 (spin mutex:callout) 23 569860 423 88509 6 0 51 100 /usr/src/sys/kern/subr_taskqueue.c:71 (spin mutex:fast_taskqueue) 46 489089 188 90306 5 0 6 7 /usr/src/sys/kern/subr_sleepqueue.c:232 (spin mutex:sleepq chain) 102 488839 28464 19935 24 1 15840 5849 /usr/src/sys/dev/em/if_em.c:1563 (sleep mutex:em1) 70692 443077 0 24 18461 0 0 0 /usr/src/sys/sys/buf.h:280 (lockmgr:bufwait) 61 291437 6501 8148 35 0 5664 1610 /usr/src/sys/dev/em/if_em.c:1563 (sleep mutex:em0) 27 2760115 474506 1346693 2 0 102015 137670 /usr/src/sys/dev/em/if_em.c:949 (sleep mutex:em1) 246691 246691 0 1 246691 0 0 0 /usr/src/sys/netinet/tcp_timer.c:423 (sleep mutex:tcp) 13 121639 10 60134 2 0 0 2 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock 0) 13 119466 1 60135 1 0 0 1 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock 3) 9 111044 5 60134 1 0 0 1 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock 1) 107 107 246687 1 107 246687 0 1 /usr/src/sys/netinet/tcp_timer.c:438 (sleep mutex:inp) you can see the whole file here - http://89.186.204.158/profiling.txt -- Best Wishes, Stefan Lambrev ICQ# 24134177Received on Sun Jan 27 2008 - 08:30:35 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:26 UTC