Re: Interrupt routine usage not shown by top in 8.0

Barney Cordoba wrote:
> 
> 
> 
> --- On Tue, 3/17/09, Sam Leffler <sam_at_freebsd.org> wrote:
> 
>> From: Sam Leffler <sam_at_freebsd.org>
>> Subject: Re: Interrupt routine usage not shown by top in 8.0
>> To: barney_cordoba_at_yahoo.com
>> Cc: current_at_freebsd.org
>> Date: Tuesday, March 17, 2009, 4:41 PM
>> Barney Cordoba wrote:
>>>
>>> --- On Tue, 3/17/09, Robert Watson
>> <rwatson_at_FreeBSD.org> wrote:
>>>   
>>>> From: Robert Watson <rwatson_at_FreeBSD.org>
>>>> Subject: Re: Interrupt routine usage not shown by
>> top in 8.0
>>>> To: "Paolo Pisati"
>> <p.pisati_at_oltrelinux.com>
>>>> Cc: "Barney Cordoba"
>> <barney_cordoba_at_yahoo.com>, current_at_freebsd.org
>>>> Date: Tuesday, March 17, 2009, 11:24 AM
>>>> On Tue, 17 Mar 2009, Paolo Pisati wrote:
>>>>
>>>>     
>>>>> perhaps i misunderstood your question, but
>> i'll
>>>>>       
>>>> try to explain a bit:
>>>>     
>>>>> before 7.0, bus_setup_intr() took just one
>> function
>>>>>       
>>>> thus you could have an INTR_FAST or an INTR_MPSAFE
>> handler,
>>>> and you choose the kind of handler via a flag
>> (INTR_FAST in
>>>> this case).
>>>>     
>>>>> after 7.0, bus_setup_intr() took 2 functions,
>> thus you
>>>>>       
>>>> could have: a fast handler (aka filter), or an
>> ithread
>>>> handler (aka mpsafe), or a fast + ithread handler
>> (available
>>>> only with INTR_FILTER turned on).
>>>>     
>>>>> in bus_setup_intr() the first function pointer
>> is for
>>>>>       
>>>> the filter side of the handler, while the second
>> pointer is
>>>> for the ithread part, and if you declare both you
>> can filter
>>>> events (interrupts) and call the rest of the
>> device driver
>>>> (the ithread part) after the filter has recognized
>> and
>>>> acknowledged&masked the interrupt.
>>>>
>>>> This clarifies my misunderstanding, thanks!
>>>>
>>>>     
>>> I'd still be interested in knowing the specific
>> advantage/consequences
>>> of a fast filter vs an MPSAFE ithread?
>>>
>>> In what circumstance would using a filter and then
>> launching a task be advantageous over just using an ithread?
>>>   
>> It mostly depends on the hardware (unless the fast handler
>> does actual work).  If ack'ing the interrupt improves
>> latency (e.g. by allowing the device to do other things)
>> then it's better to do that in the filter method even if
>> the actual work is deferred to the ithread.  It's also
>> important when interrupts are not edge-triggered; you want
>> to shut them up asap.
>>
>> So, what device are you doing a driver for?
>>
>>    Sam
> 
> I'm working with the igb and ixgbe drivers. Neither of them
> use the filters, but em does. Since they are all maintained
> by the same person I was curious as to why the em used filters
> and the igb and ixgbe, which are supposedly higher performance
> cards, use MPSAFE ithreads. 
> 
> Barney
> 

Filters were introduced into the em driver to get around a problem in
certain Intel chipsets that caused aliased interrupts.  That's a
different topic of discussion that you are welcome to search the mail
archives on.  The filter also solves performance and latency problems
that are inherent to the ithread model when interrupts are shared
between multiple devices.  This is especially bad when a high speed
device like em shares an interrupt with a low speed device like usb.
In the course of testing and validating the filter work, I found that
filters caused no degradation in performance or excess context switches,
while cleanly solving the above two problems that were common on
workstation and server class machines of only a few years ago.

However, both of these problems stemmed from using legacy PCI
interrupts.  At the time, MSI was still very new and very unreliable.
As the state of the art progressed and MSI became more reliable, its
use has become more common and is the default in several drivers.  The
igb and ixgbe drivers and hardware both prefer MSI over legacy
interrupts, while the em driver and hardware still has a lot of legacy
hardware to deal with.  So when MSI is the common/expected/default case,
there is less of a need for the filter/taskqueue method.

Filters rely on the driver being able to reliably control the interrupt
enable state of the hardware.  This is possible with em hardware, but
not as reliable with bge hardware, so the stock driver code does not
have it implemented.  I am running a filter-enabled bge driver in
large-scale production, but I also have precise control over the
hardware being used.  I also have filter patches for the bce driver, but
bce also tends to  prefer MSI, so there isn't a compelling reason to
continue to develop the patches.


Scott