Re: ixgbe and fast interrupts

On Friday, November 18, 2011 12:06:15 pm Luigi Rizzo wrote:
> On Fri, Nov 18, 2011 at 08:00:06AM -0500, John Baldwin wrote:
> > On Friday, November 18, 2011 3:46:02 am Matteo Landi wrote:
> > > > you probably want to be using MSI-X for a 10G NIC instead of INTx anyway.
> > > 
> > > Why do you say that? Is MSI-X faster than INTx in terms of interrupt
> > > latency? When should I use MSI-X, instead of fast filters interrupts
> > > (fast interrupt?), instead of ithread interrupts? Thanks in advace.
> > 
> > With MSI-X you can have more than one interrupt and those interrupts can be 
> > distributed across CPUs.  This means you can (somewhat) tie each queue on your 
> > NIC to a different CPU.
> > 
> > MSI-X vs INTx is orthogonal to fast vs filter, but in general MSI and MSI-X 
> > interrupts are not shared, and require no interrupt masking in hardware (they 
> > are effectively edge-triggered), so using a filter for MSI is rather pointless 
> > and only adds needless complexity.  For MSI I would just use a theraded 
> > interrupt handler.  For INTx, I would only use a fast interrupt handler if 
> > there is a really good reason to do so (e.g. em(4) does so to work around 
> > broken Intel Host-PCI bridges).
> 
> A bit more context: Matteo is looking at the latency of RPCs across
> the network involving userspace processes, and possibly using the
> netmap API. As we understand it:
> 
> if you are not using a filter, the interrupt calls a "predefined"
> filter (kern_intr.c::intr_event_schedule_thread() ? ) which wakes
> up the handler thread which in turn wakes up the user process.  This
> means two scheduler invocations on each side.

Yes, but if you use a filter you still have to do that as your filter would
just be queueing a task to on a taskqueue which would then do the actual
selwakeup() from a taskqueue thread.  Filters are typically used to avoid
masking the interrupt in the PIC, or to limit the handlers executed on a
shared interrupt.

> In the case of netmap, all the handler needs to do is a selwakeup()
> of the user thread blocked on the file descriptor, so if this
> can be done in the filter we can save an extra step through the
> scheduler.

You can't call selwakeup() from a filter, it is not safe since it uses
mutexes, etc.  There are only a few things you can do from a filter.
You could do a plain wakeup() if you let userland use a custom ioctl to
block on the filter, but not selwakeup().

-- 
John Baldwin