Re: sysctl kern.ipc.somaxconn limit 65535 why?

On Wed, 4 Jan 2012, Chuck Swiger wrote:

> On Jan 4, 2012, at 1:49 PM, Arnaud Lacombe wrote:
>> Hi,
>>
>> On Wed, Jan 4, 2012 at 4:42 PM, Chuck Swiger <cswiger_at_mac.com> wrote:
>>> On Jan 4, 2012, at 1:03 PM, Dan The Man wrote:
>>>>> However, I'm not convinced that it is useful to do this.  At some point, you are better off timing out and retrying via exponential backoff than you are queuing hundreds of thousands of connections in the hopes that they will eventually be serviced by something sometime considerably later.
>>>>
>>>> I agree completely, in practical application this makes sense, but why should the OS dictate not being able to temporarily set that setting higher in order to fully benchmark the application at 100k+ in the listen queue if the developer so chooses? I think that alone should be a good reason, to make freebsd developer friendly.
>>>
>>> The job of the OS is to manage resources on behalf of the users and processes using the system.
>>
>> No. The job of the OS is to service the user with the resource
>> available, not constrict the user within some arbitrary predefined
>> wall when there is still plenty of room available. If resource become
>> scarce, then take action.
>
> It is not arbitrary.  Systems ought to provide sensible limits, which can be adjusted if needed and appropriate.  The fact that a system might have 50,000 file descriptors globally available does not mean that it would be OK for any random process to consume half of them, even if there is still adequate room left for other tasks.  It's common for "ulimit -n" to be set to 256 or 1024.

Sensibly limits means a sensible stock default, not imposing an OS limit 
on what admin/developer can set on his own hardware.

With the new IBM developments underway of 16 core atom 
processors and hundreds of gigabytes of memory, surely a backlog of 100k 
is manageable. Or what about the future of 500 core systems with a 
terrabyte of memory, 100k listen queue could be processed instantly.

>
>>> Some developers feel that VM means that the system should always claim have more memory available, but always saying "yes" isn't "managing resources".  I'd rather have the OS return a null pointer and set ENOMEM when someone tries to malloc() more memory than the system (including swap, VM overcommit, etc) has, and I expect developers to code well enough to handle malloc() failures.
>>
>> this is merely a policy issue, not yours to impose.
>
> If we're speaking of machines which I administer, it is a policy issue that would be mine to impose.
> If we're speaking of someone else's machines, then they can set their own policies as they please.
>
>>> Setting the listen queue to an arbitrarily high value isn't useful, and developers would be better advised to pay attention to best practices in the face of a massive connection backlog.
>>
>> Stress-testing isn't about "best practice". It is about shaking enough
>> the system to highlight weak point.
>
> Yes.  If the system doesn't handle connectivity problems via something like exponential backoff, then the weak point is poor software design and not FreeBSD being unwilling to set the socket listen queue to a value in the hundreds of thousands.
>

I think what me and Arnaud are trying to say here, is let freebsd use a 
sensible default value, but let the admin dictate the actual policy if he 
so chooses to change it for stress testing, future proofing or anything 
else.

Dan.


--
Dan The Man
CTO/ Senior System Administrator
Websites, Domains and Everything else
http://www.SunSaturn.com
Email: Dan_at_SunSaturn.com