Re: Potential deadlock on mbuf

On Tue, 3 Apr 2012, Andre Oppermann wrote:

>> On 02.04.2012 18:21, Alexandre Martins wrote:
>>> Dear,
>>> 
>>> I have currently having troubles with a basic socket stress.
>>> 
>>> The socket are setup to use non-blocking I/O.
>>> 
>>> During this stress-test, the kernel is running mbuf exhaustion, the goal is 
>>> to
>>> see system limits.
>>> 
>>> If the program make a write on a socket during this mbuf exhaustion, it 
>>> become
>>> blocked in "write" system call. The status of the process is "zonelimit" 
>>> and
>>> whole network I/O fall in timeout.
>>> 
>>> I have found the root cause of the block  :
>>> http://svnweb.freebsd.org/base/head/sys/kern/uipc_socket.c?view=markup#l1279
>>> 
>>> So, the question is : Why m_uiotombuf is called with a blocking parameter
>>> (M_WAITOK) even if is for a non-blocking socket ?
>>> 
>>> Then, if M_NOWAIT is used, maybe it will be usefull to have an 'ENOMEM' 
>>> error.
>
> I'm surprised you can even see blocking of malloc(... M_WAITOK).
> O_NONBLOCK is mostly for operations that might block for a long time,
> but malloc() is not expected to block for long.  Regular files are
> always so non-blocking that most file systems have no references to
> O_NONBLOCK (or FNONBLOCK), but file systems often execute memory
> allocation code that can easily block for as long as malloc() does.
> When malloc() starts blocking for a long time, lots of things will
> fail.

The fact is that all mbuf are used by connected sockets, waiting
for program reading. But the program try to make a write
for data transfert.
So, the mbuf allocation block in waiting of available mbuf,
but the only proccess wich can "free" mbuf is blocked.
The mbuff allocation is deadlocked and host become unreachable.

>> This is a bit of an catch-22 we have here.  Trouble is that when
>> we return with EAGAIN the next select/poll cycle will tell you
>> that this and possibly other sockets are writeable again, when in
>> fact they are not due to kernel memory shortage.  Then the application
>> will tightly loop around the "writeable" non-writeable sockets.
>> It's about the interaction of write with O_NONBLOCK and select/poll
>> on the socket.
>
> This would be difficult to handle better.

I play with the flag. I switched it to M_NOWAIT en return a EAGAIN error
if allocation failed.
The program fail some write, but try again later and the host continue
to be reachable.
I agree that solution is not correct.

>> Do you have any references how other OSes behave, in particular
>> Linux?
>>
>> I've added bde_at_ as our resident standards compliance expert.
>> Hopefully he can give us some more insight on this issue.
>
> Standards won't say what happens at this level of detail.
>
> Blocking for network i/o is still completely broken at levels below
> sockets AFAIK.  I (and ttcp) mainly wanted it to work for send() of
> udp.  I saw no problems at the socket level, but driver queues just
> filled up and send() returned ENOBUFS.  I wanted either the opposite
> of O_NONBLOCK (block until !ENOBUFS), or at least for select() to work
> for waiting until !ENOBUFS.  But select() doesn't work at all for this.
> It seemed to work better in Linux.
>
> Bruce

Alexandre Martins