Re: Potential source of interrupt aliasing

On Sun, 10 Apr 2005, John Baldwin wrote:

> > For fun, I put together this one-liner to at least test the theory:
> >
> > http://people.freebsd.org/~dwhite/intr_machdep.c.20050409.patch
>
> Doing this should result in an instant interrupt storm since your PCI
> interrupts (which are level triggered) should now keep firing over and over
> and the machine will spend all it's time rescheduling the ithread.  If it's
> not doing that then this is very curious.

Causing the storm was partly the point. I wanted to find a way to weather
it as cheaply as possible until the ithread would get run.

> > This makes the aliased interrupts go away at the cost of hitting
> > sched_lock for each ithread interrupt.  I'm trying to dream up a way of
> > telling if the ithread is active or scheduled so we don't need to bump it
> > again with setrunqueue().  I think I can use the it_need sparkplug to at
> > least save mutex grabs on interrupts that fire much faster than their
> > ithread handlers cycle.
> >
> > I'd like better granularity, though, but I'm not sure that setting a flag
> > that is cleared just before mi_switch() in ithread_loop() and set just
> > after won't race against an interrupt checking it to see if it needs to
> > setrunqueue() the ithread. Maybe clear the flag before trying to grab
> > sched_lock?
>
> it_need already does this.  You haven't added any extra sched_locks.  What've
> you done is change the interrupt code to never mask interrupt sources.  If
> you aren't getting an interrupt storm on your first PCI interrupt then that
> is very curious indeed.  Have you tried this with SMP disabled so that you
> only have one CPU running?  It may be that due to SMP your ithread still gets
> a chance to run on one CPU while the other CPU is stormed with interrupt
> requests.  Note that since storm detection is done in the ithread, this type
> of storm won't be detected and throttled.  It also might not show up in the
> vmstats since we don't use atomic ops there anymore depending on whether or
> not the interrupt bounces between CPUs that keep overwriting the count on top
> of each other.

I think you're right in the SMP case -- if one CPU gets bogged down
another can get the ithread to bail it out.

For the UP case, turning on INVARIANTS+WITNESS+!WITNESS_SKIPSPIN causes a
lockup as the IOAPIC vectors are unmasked before the other CPUs are
started on Scott's WV2. A traceback shows softclock's ithread running and
taking an interrupt from em1, which is assumably triggering over and over
since softclock can't seem to get out of its ithread. The debugging
options and the constant interrupts slow it down so much that it isn't
cycling fully before softclock fires again (or thats what it looks like).
Adding a mi_switch() in the callout_lock drop/pickup didn't seem to help.

Adding a bypass of setrunqueue() in ithread_schedule() if it_need is set
also had no impact.

I haven't tried things with PREEMPTION.

I'm also taking suggestions for other places to try to mask interrupts. I
pondered pulling the vector from the IDT, but that generates a trap. I
don't think there's anything we can do in the LAPIC thats anything less
than disabling interrupts entirely.

-- 
Doug White                    |  FreeBSD: The Power to Serve
dwhite_at_gumbysoft.com          |  www.FreeBSD.org