kqueue disable on delivery...

I have implemented a couple additional features to kqueue.  These allow
kqueue to be a multithreaded event delivery system that can guarantee
that the event will only be active in one thread at any time.

The first is EV_DOD, aka disable on delivery.  When the event will be
delivered to userland, the knote is marked disabled so we don't
have to go through the expense of reallocing the knote each time.
(Reallocation of the knote is also lock intensive, and disabling is
cheap.)

Even though this means that the event will only ever be active in a
thread at a time, (when you're done handling the event, you reenable
it), removing the event from the queue outside the event handler (say
a timeout handler for the connection) poses to be a problem.  If you
simply close the socket, the event disappears, but then there is a
race between another event being created with the same socket, and
notification of the handler that you want the event to stop.

In order to handle that situation, I have come up w/ EV_FORCEOS, aka
FORCE ONE_SHOT.  EV_ONESHOT events have the advantage that once queued,
they don't care if they have been activated or not, they will be returned
the next round.  This means that the timeout handler can safely set
EV_FORCEOS on the handler, and either if it's _DISABLED (handler running
and will reenable it), or it's _ENABLED, it will get dispatched, allowing
the handler to detect the EV_FORCEOS flag and teardown the connection.

I have expanded a custom web server to make use of these features and
so it's been stable.  I believe most of the bugs are concurency bugs
going from a single threaded kqueue design, to allowing multiple threads
access to some of the data structures.

The patch is available at:
http://people.FreeBSD.org/~jmg/kqueue.dod.patch

As part of the patch, I added more strict checking of the flags, so
that unknown flags will return an EINVAL.

Comments?  Suggestions?

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."