Re: sysctl kern.ipc.somaxconn limit 65535 why?

On Wed, Oct 3, 2012 at 11:45 AM,  <freebsd_at_chrysalisnet.org> wrote:
> Hi everyone.
>
> Actually 65k sockets is incredibly easy to reach.
>
> I manage some servers for a very large website, it currently has several
> http servers clustered to handle daily traffic and this is only dynamic
> content, static has its own servers, databases also have own servers.
>
> We recently started using memcached to cache some queries and we found on
> default server limits sockets were saturated extremely quickly, on the linux
> servers bumping it to a few hundred K solved the issue, wasnt possible on
> bsd.  We did manage to get it down to only needing about 20k by setting
> extremely low timeout values.
>
> In addition we had to migrate all our mysql servers from freebsd to debian
> because they were hitting some arbitary OS limit but I could never figure
> out what, sys% usage went through the roof when this limit was hit, issue
> didnt occur on debian.  I feel recently freebsd is more focused on desktop's
> and as such developer's never develop for a heavy server usage scenario, and
> I keep coming across hardcoded low limits.  As rightly pointed out default
> values now days are useless 128 for somaxconn? maybe ok for a desktop.
>
> Freebsd also has issues with high numbers of cpu cores, very high locking
> overheads,  the problem is the modern hardware is focusing on more and more
> cores and as such much more work on a server so a OS has to handle more and
> more processing and data, this includes more bigger connection queues and so
> on.  FreeBSD just isnt scaling with more powerful hardware.  Also forgetting
> as well DDOS attack scenarios where a high queue size is useful.
>
> I have used FreeBSD since the 4.x days and have been forced to migrate OS on
> several servers as FreeBSD to be blunt couldnt handle the load on those
> servers.  It does seem as if developers of the OS are getting out of touch
> in respect to modern hosting enviromnents.  I cant tell app developers to
> fix their apps to work on FreeBSD, they dont care, if it works fine on
> windows and linux then the app isnt broken as far as they are concerned.
> The more FreeBSD dev's have this aatitude of not adapting the less machines
> will run the OS.  I hate many things about debian but at the end of the day
> I have to use what can handle the load or I lose my job.
>
> I do accept its entirely possible some tunables could fix any issues I have
> come across but believe me I have spent 100s of man hours on issues, and
> tuned everything I could find.
>
> To me this 32767 limit on somaxconn seems restrive and I vote for it been
> bumped to at least 8x that amount as a minimum.  A more suitable default
> would probably be around 1024 as well instead of 128.  Dead lingering
> connections in timewait etc. all use one of these, and that can very easily
> get into the 1000s.

Here's where it's being held at 65535 (sys/kern/kern_uipc.c):

3276 static int
3277 sysctl_somaxconn(SYSCTL_HANDLER_ARGS)
3278 {
3279         int error;
3280         int val;
3281
3282         val = somaxconn;
3283         error = sysctl_handle_int(oidp, &val, 0, req);
3284         if (error || !req->newptr )
3285                 return (error);
3286
3287         if (val < 1 || val > USHRT_MAX)
3288                 return (EINVAL);
3289

More details about the need for this limit are in
http://svnweb.freebsd.org/base?view=revision&revision=140730 .

It looks like this needs to be enhanced to support more than
USHRT_MAX, which will require socket structure changes and other fun
things, but should be possible... I've CCed glebius for comment on
this.

Thanks!
-Garrett