Re: (boost::)asio and kqueue problem

On Wed, Jul 20, 2016 at 07:38:09AM +0300, Konstantin Belousov wrote:

> On Tue, Jul 19, 2016 at 05:35:59PM +0200, Hartmut Brandt wrote:
> > Hi,
> > 
> > I'm trying to use asio (that's boost::asio without boost) to handle 
> > listening sockets asynchronuosly. This appears not to work. There are also
> > some reports on the net about this problem. I was able to reproduce the 
> > problem with a small C-programm that does the same steps as asio. The 
> > relevant sequence of system calls is:
> > 
> > kqueue()					 = 3 (0x3)
> > socket(PF_INET,SOCK_STREAM,6)			 = 4 (0x4)
> > setsockopt(0x4,0xffff,0x800,0x7fffffffea2c,0x4)	 = 0 (0x0)
> > kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0)
> > setsockopt(0x4,0xffff,0x4,0x7fffffffea2c,0x4)	 = 0 (0x0)
> > bind(4,{ AF_INET 0.0.0.0:8080 },16)		 = 0 (0x0)
> > listen(0x4,0x80)				 = 0 (0x0)
> > ioctl(4,FIONBIO,0xffffea2c)			 = 0 (0x0)
> > kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0)
> > kevent(3,0x0,0,0x7fffffffe5a0,32,0x0)		 ERR#4 'Interrupted system call'
> > 
> > The problem here is that asio registers each file descriptor with 
> > EVFILT_READ and EVFILT_WRITE as soon as it is opened (first kevent call). 
> > After bringing the socket into the listening state and when async_accept() 
> > is called it registers the socket a second time. According to the man page 
> > this is perfectly legal and can be used to modify the registration.
> > 
> > With this sequence of calls kevent() does not return when a connection is 
> > established successfully.
> > 
> > I tracked down the problem and the reason is in soo_kqfilter(). This is 
> > called for the first EVFILT_READ registration and decides based on the 
> > SO_ACCEPTCONN flag which filter operations to use solisten_filtops or 
> > soread_filtops. In this case it chooses soread_filtops.
> > 
> > The second EVFILT_READ registration does not call soo_kqfilter() again, 
> > but just updates the filter from the data and fflags field so the 
> > listening socket ends up with the wrong filter operations.
> > 
> > The attached patch fixes this (kind of) by using the f_touch 
> > operation (currently used only by the user filter). The filt_sotouch() 
> > function changes the operation pointer in the knote when the socket is now 
> > in the listening state. I suppose that the required locking is already 
> > done in kqueue_register(), but I'm not sure. Asynchronous accepting now works.
> > 
> > A better fix would probably be to change the operation vector on all 
> > knotes attached to the socket in solisten(), but I fear I don't have the 
> > necessary understanding of the locking that is required for this.
> > 
> > Could somebody with enough kqueue() knowledge look whether the patch is 
> > correct lock-wise?
> I find it weird that the fix still requires re-registration of the socket
> event to get it working after socket is marked as listen.  In other words,
> until the re-registration is done, the events for the registered filter
> are lost.
> 
> IMO more correct solution would be to merge the filt_solisten and
> filt_soread, deciding which path to take by testing the SO_ACCEPTCON
> flag in the f_event() op.
> 
> This should also eliminate the concerns with the patching of pointer
> which is not supposed to be changed.

may be this is related to netmap problem also?