Re: Interrupt routine usage not shown by top in 8.0

--- On Wed, 3/18/09, Scott Long <scottl_at_samsco.org> wrote:

> From: Scott Long <scottl_at_samsco.org>
> Subject: Re: Interrupt routine usage not shown by top in 8.0
> To: barney_cordoba_at_yahoo.com
> Cc: "Sam Leffler" <sam_at_freebsd.org>, current_at_freebsd.org
> Date: Wednesday, March 18, 2009, 1:34 AM
> Barney Cordoba wrote:
> > 
> > 
> > 
> > --- On Tue, 3/17/09, Sam Leffler
> <sam_at_freebsd.org> wrote:
> > 
> >> From: Sam Leffler <sam_at_freebsd.org>
> >> Subject: Re: Interrupt routine usage not shown by
> top in 8.0
> >> To: barney_cordoba_at_yahoo.com
> >> Cc: current_at_freebsd.org
> >> Date: Tuesday, March 17, 2009, 4:41 PM
> >> Barney Cordoba wrote:
> >>>
> >>> --- On Tue, 3/17/09, Robert Watson
> >> <rwatson_at_FreeBSD.org> wrote:
> >>>   
> >>>> From: Robert Watson
> <rwatson_at_FreeBSD.org>
> >>>> Subject: Re: Interrupt routine usage not
> shown by
> >> top in 8.0
> >>>> To: "Paolo Pisati"
> >> <p.pisati_at_oltrelinux.com>
> >>>> Cc: "Barney Cordoba"
> >> <barney_cordoba_at_yahoo.com>,
> current_at_freebsd.org
> >>>> Date: Tuesday, March 17, 2009, 11:24 AM
> >>>> On Tue, 17 Mar 2009, Paolo Pisati wrote:
> >>>>
> >>>>     
> >>>>> perhaps i misunderstood your question,
> but
> >> i'll
> >>>>>       
> >>>> try to explain a bit:
> >>>>     
> >>>>> before 7.0, bus_setup_intr() took just
> one
> >> function
> >>>>>       
> >>>> thus you could have an INTR_FAST or an
> INTR_MPSAFE
> >> handler,
> >>>> and you choose the kind of handler via a
> flag
> >> (INTR_FAST in
> >>>> this case).
> >>>>     
> >>>>> after 7.0, bus_setup_intr() took 2
> functions,
> >> thus you
> >>>>>       
> >>>> could have: a fast handler (aka filter),
> or an
> >> ithread
> >>>> handler (aka mpsafe), or a fast + ithread
> handler
> >> (available
> >>>> only with INTR_FILTER turned on).
> >>>>     
> >>>>> in bus_setup_intr() the first function
> pointer
> >> is for
> >>>>>       
> >>>> the filter side of the handler, while the
> second
> >> pointer is
> >>>> for the ithread part, and if you declare
> both you
> >> can filter
> >>>> events (interrupts) and call the rest of
> the
> >> device driver
> >>>> (the ithread part) after the filter has
> recognized
> >> and
> >>>> acknowledged&masked the interrupt.
> >>>>
> >>>> This clarifies my misunderstanding,
> thanks!
> >>>>
> >>>>     
> >>> I'd still be interested in knowing the
> specific
> >> advantage/consequences
> >>> of a fast filter vs an MPSAFE ithread?
> >>>
> >>> In what circumstance would using a filter and
> then
> >> launching a task be advantageous over just using
> an ithread?
> >>>   
> >> It mostly depends on the hardware (unless the fast
> handler
> >> does actual work).  If ack'ing the interrupt
> improves
> >> latency (e.g. by allowing the device to do other
> things)
> >> then it's better to do that in the filter
> method even if
> >> the actual work is deferred to the ithread. 
> It's also
> >> important when interrupts are not edge-triggered;
> you want
> >> to shut them up asap.
> >>
> >> So, what device are you doing a driver for?
> >>
> >>    Sam
> > 
> > I'm working with the igb and ixgbe drivers.
> Neither of them
> > use the filters, but em does. Since they are all
> maintained
> > by the same person I was curious as to why the em used
> filters
> > and the igb and ixgbe, which are supposedly higher
> performance
> > cards, use MPSAFE ithreads. 
> > 
> > Barney
> > 
> 
> Filters were introduced into the em driver to get around a
> problem in
> certain Intel chipsets that caused aliased interrupts. 
> That's a
> different topic of discussion that you are welcome to
> search the mail
> archives on.  The filter also solves performance and
> latency problems
> that are inherent to the ithread model when interrupts are
> shared
> between multiple devices.  This is especially bad when a
> high speed
> device like em shares an interrupt with a low speed device
> like usb.
> In the course of testing and validating the filter work, I
> found that
> filters caused no degradation in performance or excess
> context switches,
> while cleanly solving the above two problems that were
> common on
> workstation and server class machines of only a few years
> ago.
> 
> However, both of these problems stemmed from using legacy
> PCI
> interrupts.  At the time, MSI was still very new and very
> unreliable.
> As the state of the art progressed and MSI became more
> reliable, its
> use has become more common and is the default in several
> drivers.  The
> igb and ixgbe drivers and hardware both prefer MSI over
> legacy
> interrupts, while the em driver and hardware still has a
> lot of legacy
> hardware to deal with.  So when MSI is the
> common/expected/default case,
> there is less of a need for the filter/taskqueue method.
> 
> Filters rely on the driver being able to reliably control
> the interrupt
> enable state of the hardware.  This is possible with em
> hardware, but
> not as reliable with bge hardware, so the stock driver code
> does not
> have it implemented.  I am running a filter-enabled bge
> driver in
> large-scale production, but I also have precise control
> over the
> hardware being used.  I also have filter patches for the
> bce driver, but
> bce also tends to  prefer MSI, so there isn't a
> compelling reason to
> continue to develop the patches.
> 
> 
> Scott

Assuming same technique is used within an ithread as with a fast 
interrupt, that is:

filtered_foo(){
   taskqueue_enqueue();
   return FILTER_HANDLED;
}

ithread_foo(){
   taskqueue_enqueue();
   return;
}

Is there any additional overhead/locking in the ithread method? I'm
looking to get better control over cpu distribution.

Barney