Re: hacking - aio_sendfile()

On Thu, Jul 11, 2013 at 12:04:57PM -0700, Scott Long wrote:
> 
> On Jul 11, 2013, at 11:48 AM, Konstantin Belousov <kostikbel_at_gmail.com> wrote:
> 
> > On Thu, Jul 11, 2013 at 11:44:32AM -0700, Scott Long wrote:
> >> 
> >> On Jul 10, 2013, at 11:17 PM, Konstantin Belousov <kostikbel_at_gmail.com> wrote:
> >> 
> >>> On Wed, Jul 10, 2013 at 04:36:23PM -0700, Adrian Chadd wrote:
> >>>> Hiya,
> >>>> 
> >>>> I've started writing an aio_sendfile() syscall.
> >>>> 
> >>>> http://people.freebsd.org/~adrian/ath/20130710-aio-sendfile-3.diff
> >>>> 
> >>>> Yes, the diff is against -HEAD and not stable/9.
> >>>> 
> >>>> It's totally horrible, hackish and likely bad. I've only done some
> >>>> very, very basic testing to ensure it actually works; i haven't at all
> >>>> stress tested it out yet. It's also very naive - I'm not at all doing
> >>>> any checks to see whether I can short-cut to do the aio there and
> >>>> then; I'm always queuing the sendfile() op through the worker threads.
> >>>> That's likely stupid and inefficient in a lot of cases, but it at
> >>>> least gets the syscall up and working.
> >>> Yes, it is naive, but for different reason.
> >>> 
> >>> The kern_sendfile() is synchronous function, it only completes after
> >>> the other end of the network communication allows it. This means
> >>> that calling kern_sendfile() from the aio thread blocks the thread
> >>> indefinitely by unbounded sleep.
> >> 
> >> 
> >> No, kern_sendfile is async unless you specify the SF_SYNC hack flag.
> >> Otherwise, it'll fill the socket buffer and then return immediately, unless
> >> the socket buffer is full and the socket is set to blocking mode.  That's
> >> outside the scope, as I said in my previous email.
> > 
> > You do not understand what I said, please re-read both my mail and code
> > before replying.  Implementing aio_sendfile() as proposed would create
> > yet another possibility of indefinitely block all processes using aio.
> 
> I'm lost, maybe I missed some emails?  I see a set of emails where you incorrectly
> state that kern_sendfile() will always call sbwait(), and then you backtrack on that
> and claim that it's unacceptable to enforce that SS_NBIO be used for aio operations.
> I apologize if I'm missing something here.

Can you cite my exact text where I claimed that kern_sendfile() always calls
sbwait ?

I wrote about this explicitely, stating that it is very easy to make
kern_sendfile() sleep for the socket buffer space, and the duration
of the sleep is user-controllable.  As result, it allows to hang all
processes doing aio calls, since aio thread pool is finite.  I am sorry
for retyping this and stealing your time by repeating.

Making the kern_sendfile() to behave from the aio context as if the
SS_NBIO was set on the socket contradicts the behaviour of other aio
operations. E.g. aio_read and aio_write do not perform short reads and
writes to not block the aio daemon threads (which is the cause of buggy
behaviour of existing aio syscalls on sockets).

More, I do not think that setting SS_NBIO is enough to prevent the
blocking of aio threads in kern_sendfile(). The send socket buffer
is locked exclusively by kern_sendfile(). Other thread which entered
sendfile(2) and was deliberately put to sleep on the low watermark,
still owns the so_snd sx. This means that aio threads trying to do
kern_sendfile() on this socket would be also blocked, for duration
controlled by other end.

That said, even assuming SS_NBIO is always enforced and other sleep
points are identified and worked around, the only benefit of such
implementation comparing with the direct sendfile(2) call would be
preventing the use of the calling thread context for disk i/o. FreeBSD
recently gained aio_mlock(2) which allows to get the same result in
non-hackish way.