Re: lapic_at_2k interrukts eating CPU cycles

On Wed, 22 Jun 2005, Emanuel Strobl wrote:

>> Because you have 'options APIC' in your kernel, and your running a post
>> 5.2 version of FreeBSD that has a massively improved interrupt routing
>> system in it.
>
> I've been running 5-current on that machine with "options APIC" for a 
> long time, back to when I had to include NO-MIXEDMODE to get it working 
> until I think Robeert Watson gave me some patches. But I never had lapic 
> so I was wondering...

Might have been John Baldwin.

In 5.x, timer interrupts are generated by a general purpose programmable 
timer and delivered as interrupts to one of your CPUs on an MP system, or 
your only CPU on a UP system.  Depending on your hardware, where the timer 
went varied a bit -- typically, round robin, or maybe your first CPU or 
third CPU.  The timer tick would then be broadcasted to the other CPUs 
using an inter-processor interrupt (IPI), so that all CPUs would see it 
for the purposes of timer-based preemption, profiling, statistics, and so 
on.

In 6.x, we use the timer in the CPU-local apic (LAPIC) to interrupt each 
CPU, rather than delivering all the interrupts to a particular CPU and 
forwarding them.  This lowers the IPI traffic overhead, which can be 
substantial due to synchronizing and preventing IPI deadlock.  In 
particular, spin locks are used around IPI interactions to prevent two or 
more CPUs from deadlocking by sending each other IPIs simultaneously, and 
therefore being unable to acknowledge each other (there's a whole 
literature on why and how to do this, if you google or check a decent MP 
OS text).

So in 5.x, you saw the primary timer interrupt traffic explicitly, and 
most interrupt monitoring tools didn't monitor IPI load.  In 6.x, you see 
the timer interrupt traffic on each CPU as lapic load for the CPU.  In 
6.x, we currently don't monitor IPI interrupt load, but if we did in 5.x 
and 6.x, you'd see that the IPI traffic level is much lower, making up for 
the increased LAPIC interrupt traffic.

I have a feature request in to John to add statistics gathering on IPIs, 
since he's currently reworking the interrupt paths.

Robert N M Watson