Re: ixgbe and fast interrupts

On Fri, Nov 18, 2011 at 08:00:06AM -0500, John Baldwin wrote:
> On Friday, November 18, 2011 3:46:02 am Matteo Landi wrote:
> > > you probably want to be using MSI-X for a 10G NIC instead of INTx anyway.
> > 
> > Why do you say that? Is MSI-X faster than INTx in terms of interrupt
> > latency? When should I use MSI-X, instead of fast filters interrupts
> > (fast interrupt?), instead of ithread interrupts? Thanks in advace.
> 
> With MSI-X you can have more than one interrupt and those interrupts can be 
> distributed across CPUs.  This means you can (somewhat) tie each queue on your 
> NIC to a different CPU.
> 
> MSI-X vs INTx is orthogonal to fast vs filter, but in general MSI and MSI-X 
> interrupts are not shared, and require no interrupt masking in hardware (they 
> are effectively edge-triggered), so using a filter for MSI is rather pointless 
> and only adds needless complexity.  For MSI I would just use a theraded 
> interrupt handler.  For INTx, I would only use a fast interrupt handler if 
> there is a really good reason to do so (e.g. em(4) does so to work around 
> broken Intel Host-PCI bridges).

A bit more context: Matteo is looking at the latency of RPCs across
the network involving userspace processes, and possibly using the
netmap API. As we understand it:

if you are not using a filter, the interrupt calls a "predefined"
filter (kern_intr.c::intr_event_schedule_thread() ? ) which wakes
up the handler thread which in turn wakes up the user process.  This
means two scheduler invocations on each side.

In the case of netmap, all the handler needs to do is a selwakeup()
of the user thread blocked on the file descriptor, so if this
can be done in the filter we can save an extra step through the
scheduler.

	cheers
	luigi