Re: sio: lots of silo overflows on Asus K8V with Moxa Smartio C104H/PCI

On Mon, 3 May 2004, Bruce Evans wrote:

> On Sun, 2 May 2004, M. Warner Losh wrote:
> 
> > In message: <20040502130027.O1806_at_gamplex.bde.org>
> >             Bruce Evans <bde_at_zeta.org.au> writes:
> > : No, but siointr() doesn't go near the tty layer or almost any other
> > : layer.  It uses pseudo-dma and schedules a SWI to transfer data between
> > : its buffers and the tty layer.  Fast interrupt handlers cannot call
> > : other layers because they cannot reasonable lock other layers or
> > : vice versa.  (Even the call to the scheduler to schedule a SWI in the
> > : current implementation is a layering violation.)
> >
> > So it is also safe to call the tty layer w/o giant held from the SWI?
> > I guess that was my question.  Or is the SWI wrapped in Giant when I
> > wasn't looking...
> 
> Unsafe.  SWIs are wrapped by Giant unless they are created using
> swi_add(... INTR_MPSAFE ...).  There are 26 calls to swi_add() in the
> tree, and the slow progress of Giant removal is shown by only 3 of
> these using INTR_MPSAFE.  The ones in sio are not in the 3.  The 3
> are all just for multiplexors that push down deciding whether to use
> Giant to lower layers:

Actually, it's a little more complex than that: most of the callers if
swi_add() with INTR_MPSAFE are actually via bus_setup_intr() which accepts
similar flags.  You'll find that a much larger number of interrupts are
capable of running mpsafe if you search for that.  Also, unless
debug.mpsafenet is turned on, the interrupt setup code will ignore the
INTR_MPSAFE flag for interrupts with INTR_TYPE_NET set.  Obviously, you
don't want to run with debug.mpsafenet set unless you're also running with
the network stack locking patches :-).

> - softclock: handles timeout.  CALLOUT_MPSAFE specifies that Giant
>   is not needed (or has been pushed down further) for individual
>   timeouts.
> - swi_net: calls netisrs.  NETISR_MPSAFE specifies that Giant is not
>   needed (or ...) for individual netisrs.
> - taskqueue_swi_run: calls queued MPSAFE tasks.  Non-MPSAFE tasks are
>   on another queue.

I've been working through some of the callouts in the system -- we have a
number of low-hanging fruit, such as some of the kernel log subsystem
pieces, that we might be able to handle pretty easily.  I have the NFS
server largely running without Giant (with the exception of entry into
VFS, where it has to grab Giant) and probably ought to merge those patches
sometime soon.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research