Re: machdep.cpu_idle_hlt and SMP perf?

On Tuesday 07 February 2006 17:46, Andrew Gallatin wrote:
> John Baldwin writes:
>  > On Tuesday 07 February 2006 17:15, Andrew Gallatin wrote:
>  > > John Baldwin writes:
>  > >  > On Monday 06 February 2006 17:37, Andrew Gallatin wrote:
>  > >  > > John Baldwin writes:
>  > >  > >  > On Monday 06 February 2006 14:46, Andrew Gallatin wrote:
>  > >  > >  > > Andre Oppermann writes:
>  > >  > >  > >  > Andrew Gallatin wrote:
>  > >  > >  > >  > > Why dooes machdep.cpu_idle_hlt=1 drop my 10GbE network
>  > >  > >  > >  > > rx performance by a considerable amount (7.5Gbs ->
>  > >  > >  > >  > > 5.5Gbs)?
>  > >  > >  >
>  > >  > >  > You may be seeing problems because it might simply take a
>  > >  > >  > while for the CPU to wake up from HLT when an interrupt comes
>  > >  > >  > in.  The 4BSD scheduler tries to do IPIs to wakeup any
>  > >  > >  > sleeping CPUs when it schedules a new thread, but that would
>  > >  > >  > add higher latency for ithreads than just preempting directly
>  > >  > >  > to the ithread.  Oh, you have to turn that on, it's off by
>  > >  > >  > default
>  > >  > >  > (kern.sched.ipiwakeup.enabled=1).
>  > >  > >
>  > >  > > Hmm..  It seems to be on by default.  Unfortunately, it does not
>  > >  > > seem to help.
>  > >  >
>  > >  > I'm not sure.
>  > >
>  > > One thing which really helps is disabling preemption.  If I do that,
>  > > I get 7.7Gb/sec with machdep.cpu_idle_hlt=1.  This is slightly better
>  > > than machdep.cpu_idle_hlt=0 and no PREEMPTION.
>  > >
>  > > BTW, net.isr.direct=1 in all testing.
>  >
>  > Do you have very little userland activity in this test?
>
> Essentially none.  netserver just sits in a loop, reading from the
> socket and throwing the data away.

If you disable preemption then in effect you are letting the idle CPUs pick up 
the ithread and not disturbing what is running on the non-idle CPU.  
sched_4bsd is supposed to be triggering the same behavior, except that it has 
to send an IPI to awaken the idle CPUs.  When you have idle_hlt=0, there are 
no idle CPUs, so 4bsd thinks they are all busy and preempts.  When you 
disable preemption, it just leaves the ithread on the runqueue until one of 
the idle CPUs notices the new thread in its idle loop and runs it.  When you 
have idle_hlt=1, then 4bsd doesn't preempt but sends an IPI.  It doesn't even 
try to preempt unless it thinks all CPUs are busy.

One thing disabling PREEMPTION does is that it enables some explicit 
FULL_PREEMPTION-like behavior in _mtx_unlock_sleep().  You might want to try 
#if 0'ing that code out to see if that is why having PREEMPTION off makes a 
difference.  (Ironically, having PREEMPTION on means _mtx_unlock_sleep() will 
preempt less often.)

-- 
John Baldwin <jhb_at_FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve"  =  http://www.FreeBSD.org