Re: kqueue disable on delivery...

On Sat, 23 Sep 2006, John-Mark Gurney wrote:

> Igor Sysoev wrote this message on Sat, Sep 23, 2006 at 11:40 +0400:
>> On Fri, 22 Sep 2006, John-Mark Gurney wrote:
>>
>>> Igor Sysoev wrote this message on Fri, Sep 22, 2006 at 17:25 +0400:
>>>> On Sun, 17 Sep 2006, John-Mark Gurney wrote:
>>>>
>>>>> I have implemented a couple additional features to kqueue.  These allow
>>>>> kqueue to be a multithreaded event delivery system that can guarantee
>>>>> that the event will only be active in one thread at any time.
>>>>>
>>>>> The first is EV_DOD, aka disable on delivery.  When the event will be
>>>>> delivered to userland, the knote is marked disabled so we don't
>>>>> have to go through the expense of reallocing the knote each time.
>>>>> (Reallocation of the knote is also lock intensive, and disabling is
>>>>> cheap.)
>>>>
>>>> In my opinion, it's too implementation specific flag.
>>>
>>> How else are you doing to solve having multiple threads servicing
>>> the same queue at the same time?  Also, Apple is planing on having
>>> a similar flag to EV_DOD, but I don't know what they are naming it..
>>> I've tried for a while to find out, but haven't been able to...
>>
>> As I understand EV_DOD or EV_CLEAR|EV_DOD are like simple EV_ONESHOT,
>> except the filter is not deleted on delivery, but is disabled skipping
>> some in-kernel lock overhead. That's I'd named it too implementation
>> specific.
>>
>> Yes, the EV_CLEAR|EV_DOD guarantees that the event will be active
>> in one thread only at any time. But in my practice I saw there is
>> necessity to guarantee that the socket (both events - EVFILT_READ
>> and EVFILT_WRITE) will be active in one thread only at any time.
>> It seems that is the reason why heavy threaded Solaris 10 event ports
>> use the oneshot only model where a socket is deleted from port on delivery.
>
> Only if you need to both read and write active on the socket  at once...
> In some/many servers, you only need one or the other, such as file
> transfer servers like http and ftp...

I thought about you flags and believe it would be usefull.
Also I think that the cheapness of EV_DISABLE and EV_DOD should be
documented.

>>>>> Even though this means that the event will only ever be active in a
>>>>> thread at a time, (when you're done handling the event, you reenable
>>>>> it), removing the event from the queue outside the event handler (say
>>>>> a timeout handler for the connection) poses to be a problem.  If you
>>>>> simply close the socket, the event disappears, but then there is a
>>>>> race between another event being created with the same socket, and
>>>>> notification of the handler that you want the event to stop.
>>>>>
>>>>> In order to handle that situation, I have come up w/ EV_FORCEOS, aka
>>>>> FORCE ONE_SHOT.  EV_ONESHOT events have the advantage that once queued,
>>>>> they don't care if they have been activated or not, they will be returned
>>>>> the next round.  This means that the timeout handler can safely set
>>>>> EV_FORCEOS on the handler, and either if it's _DISABLED (handler running
>>>>> and will reenable it), or it's _ENABLED, it will get dispatched, allowing
>>>>> the handler to detect the EV_FORCEOS flag and teardown the connection.
>>>>
>>>> I think it should be EVFILT_USER event, allowing to
>>>> EV_SET(&kev, fd, EVFILT_USER, 0, 0, 0, udata);
>>>> and the event should automatically sets the EV_ONESHOT flag internally.
>>>
>>> I'll agree EV_FORCEOS is open for discussion, but you did see how much
>>> code it adds right?  I was surprised at how small the patch was for the
>>> additional functionality..
>>
>> Yes, EV_FORCEOS is small patch. However, EVFILT_USER is more generic
>> (by the way, Solaris 10 event ports allow to send user-specific
>> PORT_SOURCE_USER notification).
>
> I agree EVFILT_USER would be a useful thing, but it is still different
> from EV_FORCEOS...  Would you like to contribute some the to
> EVFILT_USER?  I'll look at integrating it...

Here is patch and test program. The patch is against 6.2-PRERELEASE.
On 7.0 the EVFILT_LIO should be taked into account.

test program should show oneshot user event:
>./t
n: 1, id: 0x55, filt: -10, fl: 0x0010, ff:0, data:0x0, udata: 0x5678
n: 0, id: 0x0, filt: 0, fl: 0x0000, ff:0, data:0x0, udata: 0x0

>> Two years ago I was implementing threads for my server nginx
>> on FreeBSD 4.x, using rfork(). In the absence of EVFILT_USER I made
>> the condition variables using kill() and EV_SIGNAL and this user-level
>> code may panic kernel.
>
> Does it still?

It seems it was fixed in 1.80, 1.79.2.1, and 1.2.2.11 revisions of
src/sys/kern/kern_event.c, but there is report that is not so:
http://freebsd.rambler.ru/bsdmail/cvs-src_2005/msg04709.html
Currently nginx has threads disabled, so I could not test it.


Igor Sysoev
http://sysoev.ru/en/