Poor NFS server performance in 6.0 with SMP and mpsafenet=1

Hi,

I'm seeing incredibly poor performance when serving files from an SMP
FreeBSD 6.0RC1 server to a Solaris 10 client.  I've done some
experimenting and have discovered that either removing SMP from the
kernel, or setting debug.mpsafenet=0 in loader.conf massively improves
the speed.  Switching preemption off seems to also help.

A quick summary of my findings:  switching SMP off and setting
mpsafenet=0 gives more than 700% boost in performance.  PREEMPTION is a
pessimisation in the SMP case, mpsafenet=1 is a pessimisation in all
cases.  Given these are three of the major new features in 6.0, this
doesn't seem ideal :)

The following are average times, in seconds, to serve a 512meg file over
NFS to the Solaris host.  Each experiment was performed immediately
after a reboot of the FreeBSD host, and after one untimed transfer (in
order to warm the FreeBSD vfs cache).

No SMP, mpsafenet=1			 59.4
No SMP, mpsafenet=0			 49.4
No SMP, mpsafenet=1, no PREEMPTION	 53.1
No SMP, mpsafenet=0, no PREEMPTION	 51.9
   SMP, mpsafenet=1			351.7
   SMP, mpsafenet=0			 74.5
   SMP, mpsafenet=1, no PREEMPTION	264.9
   SMP, mpsafenet=0, no PREEMPTION	 53.7

Obviously this is not ideal, but at least I have a usable workaround.
I'm just worried that the default SMP configuration performs so poorly
(although I appreciate that GENERIC does not include SMP, anyone with a
multiprocessor machine presumably would want to use it...)

The FreeBSD machine is a twin 866MHz P3, running 6.-RC1 with ACPI
enabled.  The network cards are 64bit/66MHz em(4) cards using checksum
offloading etc.  Apart from the changes mentioned above, the kernel is
GENERIC.  I'll get a dmesg online somewhere.  I'll also recreate the
experiment, serving to a FreeBSD machine as opposed to Solaris, just to
compare.

I'm happy to do any more investigating to get to the bottom of this,
although the machines in question will be going live at the end of the
month.

Gavin