Re: 5.2-rel NFS lockup and networking performance

On Mon, 26 Jan 2004, Harald Schmalzbauer wrote:

> I just upgraded one of my servers from 5.1 to 5.2.  Now when I dd from a
> linux box on a nfs-mounted 5.2 testfile writing 100kKb takes about 5
> Minutes because the machine repeatedly "locks up". It seems every disk
> access is blocked for a while. (on the linux(debian woody) box: "dd
> if=/dev/zero of=/mnt/nfs/testfile bs=1k count=100000")  But not only
> that NFS is broken with linux (I can do the same from umy 5.2-current
> without locking up the 5.2-rel machine), also the performence is even
> more awful than with 5.1. Max transfers are about 4.5Mbyte/s and with
> samba I can get 3.5Mbyte/s. On the same machine with 5.1 it was about
> 8Mbyte/ s for NFS and about 5.5Mbyte/s for samba.  I think I'll have to
> switch back to 4.9 since I can remember having had more than 9Mbyte/s
> even for samba with much older hardware arround FreeBSD 4.4.  Can anyone
> confirm that 4.9 is still performing like this? 
> 
> Does anybody know about this NFS bug and this really weak networking
> performance? 

Regarding performance: Is this an SMP box?  If not, if you haven't
already, could you recompile your kernel without 'device apic' and
'options SMP'?  Also, there have been recent reports that a number of
older SMP systems are having interrupt storm problems with recent ACPI
drops and the new interrupt code.  If your system meets this description,
you might also try disabling ACPI.  Then rerun your tests and see what, if
any, has changed. 

Regarding the lockup: could you try compiling your kernel with 'options
BREAK_TO_DEBUGGER' -- when the system appears to freeze, hit Ctrl-Alt-Esc
on the console to drop to DDB.  If this works, the output of 'ps' would be
very helpful.  If this doesn't work, or you don't feel like typing in
pages of ps output, try hooking up a second box as a serial console --
using a serial break gets into the debugger more reliably, and you can
copy and paste. :-)  I'd also be interested in seeing stack traces for the
nfsd processes on the NFS server.  You can use "trace pid" in DDB to
generate those traces.  Also of interest is the output of "show
lockedvnods". 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research