Re: (boost::)asio and kqueue problem

On Tue, Jul 19, 2016 at 05:35:59PM +0200, Hartmut Brandt wrote:
> Hi,
> 
> I'm trying to use asio (that's boost::asio without boost) to handle 
> listening sockets asynchronuosly. This appears not to work. There are also
> some reports on the net about this problem. I was able to reproduce the 
> problem with a small C-programm that does the same steps as asio. The 
> relevant sequence of system calls is:
> 
> kqueue()					 = 3 (0x3)
> socket(PF_INET,SOCK_STREAM,6)			 = 4 (0x4)
> setsockopt(0x4,0xffff,0x800,0x7fffffffea2c,0x4)	 = 0 (0x0)
> kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0)
> setsockopt(0x4,0xffff,0x4,0x7fffffffea2c,0x4)	 = 0 (0x0)
> bind(4,{ AF_INET 0.0.0.0:8080 },16)		 = 0 (0x0)
> listen(0x4,0x80)				 = 0 (0x0)
> ioctl(4,FIONBIO,0xffffea2c)			 = 0 (0x0)
> kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0)
> kevent(3,0x0,0,0x7fffffffe5a0,32,0x0)		 ERR#4 'Interrupted system call'
> 
> The problem here is that asio registers each file descriptor with 
> EVFILT_READ and EVFILT_WRITE as soon as it is opened (first kevent call). 
> After bringing the socket into the listening state and when async_accept() 
> is called it registers the socket a second time. According to the man page 
> this is perfectly legal and can be used to modify the registration.
> 
> With this sequence of calls kevent() does not return when a connection is 
> established successfully.
> 
> I tracked down the problem and the reason is in soo_kqfilter(). This is 
> called for the first EVFILT_READ registration and decides based on the 
> SO_ACCEPTCONN flag which filter operations to use solisten_filtops or 
> soread_filtops. In this case it chooses soread_filtops.
> 
> The second EVFILT_READ registration does not call soo_kqfilter() again, 
> but just updates the filter from the data and fflags field so the 
> listening socket ends up with the wrong filter operations.
> 
> The attached patch fixes this (kind of) by using the f_touch 
> operation (currently used only by the user filter). The filt_sotouch() 
> function changes the operation pointer in the knote when the socket is now 
> in the listening state. I suppose that the required locking is already 
> done in kqueue_register(), but I'm not sure. Asynchronous accepting now works.
> 
> A better fix would probably be to change the operation vector on all 
> knotes attached to the socket in solisten(), but I fear I don't have the 
> necessary understanding of the locking that is required for this.
> 
> Could somebody with enough kqueue() knowledge look whether the patch is 
> correct lock-wise?
I find it weird that the fix still requires re-registration of the socket
event to get it working after socket is marked as listen.  In other words,
until the re-registration is done, the events for the registered filter
are lost.

IMO more correct solution would be to merge the filt_solisten and
filt_soread, deciding which path to take by testing the SO_ACCEPTCON
flag in the f_event() op.

This should also eliminate the concerns with the patching of pointer
which is not supposed to be changed.