Re: machdep.cpu_idle_hlt and SMP perf?

Andrew Gallatin wrote:
> Why dooes machdep.cpu_idle_hlt=1 drop my 10GbE network rx
> performance by a considerable amount (7.5Gbs -> 5.5Gbs)?
> 
> I've (blindly) tried leaving machdep.cpu_idle_hlt=1 enabled
> and playing with the vast array of kern.sched.ipiwakeup.* sysctls,
> but receive performance remains limited to ~5.5Gb/sec or less.
> 
> This is an 'AMD Athlon(tm) 64 X2 Dual Core Processor 3800+' running
> FreeBSD-current as of about one week ago.  The interrupt load is 
> about 22,000 device interrupts/sec (ithreaded).  Interestingly,
> the more I decrease the interrupt load by increasing the interrupt
> coalescing timer, the worse the machdep.cpu_idle_hlt=1 case does.
> 
> Is this just a case of the wakeup IPI taking a long time or blocking
> on some lock?

This may be the same problem OpenBSD has fixed last year in the handling
of the idle loop.  From the kerneltrap posting:

% "In the following article, Bob provides a first-person account of tracking
% down what began simply as a RAID performance issue, but ultimately turned
% out to be a problem with the idle loop that when fixed resulted in an
% impressive performance boost. Bob noted, "the idle loop is where the kernel
% spins when there is no work to do in userland, because of this, it's also
% where we catch and service many of our interrupts from drivers that may queue
% work to the device and then tsleep waiting for an interrupt from the card
% saying the work is done." Bob went on to explain that prior to today's fix,
% interrupts were handled appropriately when there was userland work happening,
% but not when there was nothing happening in userland and the kernel was simply
% waiting for device input/output. Read on for Bob's full account of the day,
% leading up to the discovery of the problem and the implementation of the fix,
% including performance numbers."


Here is the more specific information:

  http://kerneltrap.org/node/5169

First commit message:
  http://marc.theaimsgroup.com/?l=openbsd-cvs&m=111692513727274&w=2

The MFC with all changes in one commit message:
  http://marc.theaimsgroup.com/?l=openbsd-cvs&m=111859519015510&w=2

http://www.openbsd.org/cgi-bin/cvsweb/src/sys/arch/i386/i386/locore.s
http://www.openbsd.org/cgi-bin/cvsweb/src/sys/arch/i386/i386/apm.c


-- 
Andre