Re: [net] protecting interfaces from races between control and data ?

On 07.08.2013 09:18, Luigi Rizzo wrote:
> On Wed, Aug 7, 2013 at 5:26 AM, Mike Karels <mike_at_karels.net <mailto:mike_at_karels.net>> wrote:
>     Jumping to (near) the end of the thread, I like most of Andre's proposal.
>     Running with minimal locks at this layer is an admirable goal, and I agree
>     with most of what was said.  I have a few observations on the general changes,
>     or related issues:
>
>     There was mention of taskqueues.  I think that with MSI-X, taskqueues
>     should not be needed or used.  More specifically, having separate ithreads
>     and taskqueues, with ithreads deferring to taskqueues after some limit, makes
>     sense only for MSI and legacy interrupts.  With MSI-X, an interrupt thread
>     should be able to process packets indefinitely with sufficient CPU resources,
>     and there is no reason to context switch to a different thread periodically.
>     A periodic "yield" might be reasonable, but if it is necessary, small packet
>     performance will suffer.  However, most of this is internal to the driver.
>
>
> i am not completely clear on what is the difference between ithreads and taskqueues.

The difference between ithreads and taskqueues is actually very small and mostly in
name and how they're set up and kicked off when work is to be done.  Both are real
kernel threads.  Most of the confusion, also in Mikes response, seems to come from
the name ithreads for interrupt-threads.  However an ithread is not running in
interrupt context, only the interrupt handler (called filter) does.  Scheduling a
taskqueue from an ithread is a bit round-about but commonly done.  The bus_setup_intr(9)
man page isn't helping to clear up the confusion.

The idea is that a (legacy) interrupt is handled in two stages: 1) a small function
reading the hardware interrupt status register to determine if this hardware actually
raised an interrupt, and acknowledge it (to prevent additional interrupts from firing
while this one is handled).  If it wasn't this card, it hands off to the handler if
this interrupt line is shared; 2) the actual function/thread handling the data that
was indicated by the interrupt.  Only step 1 runs in interrupt context and thus must
be very small and avoid any form of blocking.  Step 2 is a normal kernel thread set
up as an ithread running at an elevated priority compared to user-space processes/threads.

MSI and MSI-X always are exclusive and non-shared interrupts.  Thus a handler running
in interrupt context isn't necessary for non-legacy interrupt sources.  The ithread can
be scheduled right away to do its work.

> Also, Andre's proposal requires to force-kill the ithread, but i am unclear on how to do it
> safely (i.e. without leaving the data structures in some inconsistent state), unless ithread
> periodically yields the CPU when it is in a safe state. While this is internal to the driver,
> we should probably provide some template code to avoid that each driver implements
> its own way to shutdown the ithread.

Yes, when done a well-tested, stable and performant template will be provided.

-- 
Andre