Re: zonelimit livelock, some possable workarounds

Peter Holm wrote:
> While stress testing GENERIC HEAD from Aug 26 13:08 UTC I ran into
> this livelock:
>
> http://people.freebsd.org/~pho/stress/log/cons206.html
>   
This has sadly been a common occurrence for some of us on live prod.
systems, however i think it appeared more as an application deadlock for
us as the cpu usage didn't go crazy. (IIRC)
Using our previous custom daemon that lacked good syn flood protection
on a network i help admin caused this with our own stress test suite and
once during a ddos attack.
Currently running releng_6_1 but it has happened on 5.x aswell.

Essentially what our test suite did was connect with a huge amount of
clients simultaneously and not responding to the server replay until all
clients where connected. This caused a lot of Recv-Q/Send-Q allocation
according to netstat (esp. if running the test locally), it probably ran
out of network memory and most network applications went into zoneli
state according to top.
We wont publish our test suite publicly as it is essentially a DOS
client, but can send it to a few officially listed freebsd developers
and help with replication upon request.

The workarounds we currently are trying are as follows:
* Replacing the daemons that dont handle attacks very well.
    (Some tests with the PF firewall connection rate limiting has had
have some success on a test system, but not employed live yet)
* Raising the vm.zone/mbuf_cluster indirectly with maxusers to a very
high number.
    (maxusers does not scale beyond a certain number so we had to
calculate it manually based on sysctl vm.zone and our RAM.)
    (We used to change this limit seperatly but rather have all the
limits scale with maxusers.)
* Taking a fraction of the new mbuf_cluster byte size and setting it as
a sbsize limit in login.conf so the limit cant be reached easily (or so
we hope).
* Written a small executable that attempts local network connection and
that can be used with watchdogd -e parameter to reboot the box.


/Thomas Herrlin