RE: serious networking (em) performance (ggate and NFS) problem

On Thu, 18 Nov 2004, Daniel Eriksson wrote:

> I have a Tyan Tiger MPX board (dual AthlonMP) that has two 64bit PCI
> slots.  I have an Adaptec 29160 and a dual port Intel Pro/1000 MT
> plugged into those slots. 
> 
> As can be seen from the vmstat -i output below, em1 shares ithread with
> ahc0. This is with ACPI disabled. With ACPI enabled all devices get
> their own ithread (I think, not 100% sure). However, because of some
> hardware problem (interrupt routing?), em0 interrupts will somehow leak
> into atapci1+, generating a higher interrupt load. I'm not sure how
> expensive this is. 

I see precisely this problem on several motherboards, including the Intel
Westville.  There's some speculation on the source of the problem, but I
see related problems in 4.x as well.  Either I get them on different
interrupts but both fire, or on the same interrupt.  FYI, picking the
right one depends a bit on your configuration, but generally scheduling
multiple ithreads is more expensive than running multiple handlers in the
same ithread, so I think it's generally preferable to run with them on the
same interrupt.  Especially if nothing on the same interrupt is acquiring
Giant.  Acquiring and dropping Giant uncontended is cheaper than context
switching, however.

> Finally, my question. What would you recommend:
> 1) Run with ACPI disabled and debug.mpsafenet=1 and hope that the mix of
> giant-safe and giant-locked (em and ahc) doesn't trigger any bugs. This is
> what I currently do.

This shouldn't cause bugs; the ithread handler is smart and will acquire
Giant around the ahc code.  That will also make it slower due to the extra
mutex operations, however.

> 2) Run with ACPI disabled and debug.mpsafenet=0 and accept lower network
> performance (it is a high-traffic server, so I'm not sure this is a valid
> option).
> 3) Run with ACPI enabled and debug.mpsafenet=1 and accept that em0
> interrupts "leak" to the atapci1+ ithread. This I have done in the past.

I think you want to run the ahc stuff, unfortunately.  The good news is
that the higher the load, the more interrupt mitigation/coalescing will
kick in for if_em, so the fewer you'll see.  Under load, usually my boxes
hang out at 4k-6k interrupts/sec for if_em and don't go much above that. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Principal Research Scientist, McAfee Research